[ovirt-users] host install failed when adding 4.4.6 node

I installed new ovirt node with 4.4.6 6 and the same version ovirt engine is running as stand alone manager in cent os 8. When I try to add the node to engine getting Host installation failed and below is the log from engine. node has default route and able to access the cockpit 021-05-06 20:12:49,111-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Running command: InstallVdsCommand internal: false. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2021-05-06 20:12:49,195-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:12:49,222-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDS 2021-05-06 20:12:49,222-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Before Installation host 3e4d4299-0425-4c86-9c79-930c1f43f78b, renlovkvma01.ren.broadcom.net 2021-05-06 20:12:49,227-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='Installing', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 37578a41 2021-05-06 20:12:49,234-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] FINISH, SetVdsStatusVDSCommand, return: , log id: 37578a41 2021-05-06 20:12:49,244-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Opening ssh-copy-id session on host renlovkvma01.ren.broadcom.net 2021-05-06 20:12:49,284-06 ERROR [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Can not run fence action on host 'renlovkvma01.ren.broadcom.net', no suitable proxy host was found. 2021-05-06 20:12:49,302-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_START_MIGHT_FAIL(9,010), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Start this host using Power-Management are expected to fail. 2021-05-06 20:12:49,304-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_STOP_MIGHT_FAIL(9,011), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Stop this host using Power-Management are expected to fail. 2021-05-06 20:12:49,309-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: KDUMP_DETECTION_NOT_CONFIGURED_ON_VDS(617), Kdump integration is enabled for host renlovkvma01.ren.broadcom.net, but kdump is not configured properly on host. 2021-05-06 20:12:49,323-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: USER_UPDATE_VDS(43), Host renlovkvma01.ren.broadcom.net configuration was updated by admin@internal-authz. 2021-05-06 20:12:49,577-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Executing ssh-copy-id command on host renlovkvma01.ren.broadcom.net 2021-05-06 20:12:50,424-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_ANSIBLE_INSTALL_STARTED(560), Ansible host-deploy playbook execution has started on host renlovkvma01.ren.broadcom.net. 2021-05-06 20:12:54,018-06 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Host installation failed for host '3e4d4299-0425-4c86-9c79-930c1f43f78b', 'renlovkvma01.ren.broadcom.net': null 2021-05-06 20:12:54,019-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 290371f3 2021-05-06 20:12:54,023-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] FINISH, SetVdsStatusVDSCommand, return: , log id: 290371f3 2021-05-06 20:12:54,032-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] EVENT_ID: VDS_INSTALL_FAILED(505), Host renlovkvma01.ren.broadcom.net installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details.. 2021-05-06 20:12:54,040-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-2) [d97ce48a-2ae1-4a43-bbeb-076edb686de2] Lock freed to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:14:45,462-06 INFO [org.ovirt.engine.core.bll.IsOvirtCockpitSSOStartedQuery] (default task-1) [792e24b7-7d44-460b-9204-63fa87db9b5e] The ovirt-cockpit-sso service is not running, pid file: '/var/run/ovirt-cockpit-sso/ovirt-cockpit-sso.pid'. 2021-05-06 20:16:02,350-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [424680dd] Lock Acquired to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' 2021-05-06 20:16:02,358-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [424680dd] Running command: SyncNetworkProviderCommand internal: true. 2021-05-06 20:16:02,471-06 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-1) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-05-06 20:16:02,609-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [424680dd] Lock freed to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' 2021-05-06 20:18:14,084-06 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-6) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-05-06 20:18:14,134-06 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-6) [621ba285] Running command: CreateUserSessionCommand internal: false. 2021-05-06 20:18:14,144-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [621ba285] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '10.33.184.100' using session 'dYs+hblptTN7LSrrcmfTdFOz1eekPWKZiJXnyLJW2NOfRhZF4bDnFdMX736S0xTlxe76NmzOYDl+p5qS9WAVgA==' logged in. 2021-05-06 20:18:14,873-06 INFO [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-7) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt' -- 404 2021-05-06 20:18:22,413-06 INFO [org.ovirt.engine.core.bll.IsOvirtCockpitSSOStartedQuery] (default task-6) [af0d630e-a5f8-4626-a94c-945c5477e87b] The ovirt-cockpit-sso service is not running, pid file: '/var/run/ovirt-cockpit-sso/ovirt-cockpit-sso.pid'. 2021-05-06 20:19:32,475-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2021-05-06 20:19:32,475-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engineScheduledThreadPool' is using 0 threads out of 1, 100 threads waiting for tasks. 2021-05-06 20:19:32,475-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'engineThreadMonitoringThreadPool' is using 1 threads out of 1, 0 threads waiting for tasks. 2021-05-06 20:19:33,626-06 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] Backup check started. 2021-05-06 20:19:33,630-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] EVENT_ID: ENGINE_NO_FULL_BACKUP(9,022), There is no full backup available, please run engine-backup to prevent data loss in case of corruption. 2021-05-06 20:19:33,630-06 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-49) [] Backup check completed. 2021-05-06 20:19:33,661-06 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] Attempting to update VMs/Templates Ovf. 2021-05-06 20:20:53,590-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Running command: InstallVdsCommand internal: false. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2021-05-06 20:20:53,604-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:20:53,618-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Running command: InstallVdsInternalCommand internal: true. Entities affected : ID: 3e4d4299-0425-4c86-9c79-930c1f43f78b Type: VDS 2021-05-06 20:20:53,618-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Before Installation host 3e4d4299-0425-4c86-9c79-930c1f43f78b, renlovkvma01.ren.broadcom.net 2021-05-06 20:20:53,619-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='Installing', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6e0aa36a 2021-05-06 20:20:53,623-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] FINISH, SetVdsStatusVDSCommand, return: , log id: 6e0aa36a 2021-05-06 20:20:53,628-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Opening ssh-copy-id session on host renlovkvma01.ren.broadcom.net 2021-05-06 20:20:53,677-06 ERROR [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Can not run fence action on host 'renlovkvma01.ren.broadcom.net', no suitable proxy host was found. 2021-05-06 20:20:53,679-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_START_MIGHT_FAIL(9,010), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Start this host using Power-Management are expected to fail. 2021-05-06 20:20:53,681-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_ALERT_PM_HEALTH_CHECK_STOP_MIGHT_FAIL(9,011), Health check on Host renlovkvma01.ren.broadcom.net indicates that future attempts to Stop this host using Power-Management are expected to fail. 2021-05-06 20:20:53,689-06 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: KDUMP_DETECTION_NOT_CONFIGURED_ON_VDS(617), Kdump integration is enabled for host renlovkvma01.ren.broadcom.net, but kdump is not configured properly on host. 2021-05-06 20:20:53,695-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: USER_UPDATE_VDS(43), Host renlovkvma01.ren.broadcom.net configuration was updated by admin@internal-authz. 2021-05-06 20:20:53,833-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Executing ssh-copy-id command on host renlovkvma01.ren.broadcom.net 2021-05-06 20:20:54,553-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_ANSIBLE_INSTALL_STARTED(560), Ansible host-deploy playbook execution has started on host renlovkvma01.ren.broadcom.net. 2021-05-06 20:20:57,584-06 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Host installation failed for host '3e4d4299-0425-4c86-9c79-930c1f43f78b', 'renlovkvma01.ren.broadcom.net': null 2021-05-06 20:20:57,585-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] START, SetVdsStatusVDSCommand(HostName = renlovkvma01.ren.broadcom.net, SetVdsStatusVDSCommandParameters:{hostId='3e4d4299-0425-4c86-9c79-930c1f43f78b', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 640c7f84 2021-05-06 20:20:57,589-06 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] FINISH, SetVdsStatusVDSCommand, return: , log id: 640c7f84 2021-05-06 20:20:57,592-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] EVENT_ID: VDS_INSTALL_FAILED(505), Host renlovkvma01.ren.broadcom.net installation failed. Please refer to /var/log/ovirt-engine/engine.log and log logs under /var/log/ovirt-engine/host-deploy/ for further details.. 2021-05-06 20:20:57,595-06 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-3) [82d1705d-efe4-4a91-a7d1-84fa8ed7a4d5] Lock freed to object 'EngineLock:{exclusiveLocks='[3e4d4299-0425-4c86-9c79-930c1f43f78b=VDS]', sharedLocks=''}' 2021-05-06 20:21:02,618-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-70) [2825b473] Lock Acquired to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' 2021-05-06 20:21:02,625-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-70) [2825b473] Running command: SyncNetworkProviderCommand internal: true. 2021-05-06 20:21:02,741-06 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-6) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2021-05-06 20:21:02,876-06 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-70) [2825b473] Lock freed to object 'EngineLock:{exclusiveLocks='[96aca610-8e0a-46d5-ac15-72004086325c=PROVIDER]', sharedLocks=''}' [root@renlove01 ~]# ^C [root@renlove01 ~]# cd /var/log/ovirt-engine/host-deploy/ [root@renlove01 host-deploy]# ls [root@renlove01 host-deploy]#