Hello, All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason almost simultaneously. Please help me to find the root cause. Thanks. Please note the host seems doing fine and never crash or hangs and I can migrate VMs back to it later. Here is the exact timeline of all the related events combined from the host and the VM(s): On oVirt host: /var/log/vdsm/vdsm.log: 2020-06-25 15:25:16,944-0500 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0 already removed (fileutils:54) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0 already removed (fileutils:54) [root@athos log]# journalctl -u NetworkManager --since=today -- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25 16:36:05 CDT. -- Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL /var/log/messages: Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor /var/log/libvirt/qemu/aries.log: 2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid 2713 (/usr/sbin/libvirtd) 2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown ============================================================================================= On the first VM effected (same thing on others): /var/log/ovirt-guest-agent/ovirt-guest-agent.log: MainThread::INFO::2020-06-25 15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent CredServer::INFO::2020-06-25 15:25:20,626::CredServer::262::root::CredServer has stopped. MainThread::INFO::2020-06-25 15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down. ============================================================================================= Packages version installated: Host OS version: CentOS 7.7.1908: ovirt-hosted-engine-ha-2.3.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-release43-4.3.6-1.el7.noarch ovirt-imageio-daemon-1.5.2-0.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch ovirt-imageio-common-1.5.2-0.el7.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 libvirt-4.5.0-23.el7_7.1.x86_64 libvirt-daemon-4.5.0-23.el7_7.1.x86_6 qemu-kvm-ev-2.12.0-33.1.el7.x86_64 qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64 On guest VM: ovirt-guest-agent-1.0.13-1.el6.noarch qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64
Hi Bobby, Can you please share the engine logs as well? It could help to understand what happened there. Right now, looking at the pieces of the logs you sent I couldn't spot anything unusual. Thanks in advance, On Mon, Jun 29, 2020 at 10:40 PM Bobby <bobbysch@gmail.com> wrote:
Hello,
All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason almost simultaneously. Please help me to find the root cause. Thanks.
Please note the host seems doing fine and never crash or hangs and I can migrate VMs back to it later. Here is the exact timeline of all the related events combined from the host and the VM(s):
On oVirt host: /var/log/vdsm/vdsm.log: 2020-06-25 15:25:16,944-0500 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0 already removed (fileutils:54) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0 already removed (fileutils:54)
[root@athos log]# journalctl -u NetworkManager --since=today -- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25 16:36:05 CDT. -- Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL
/var/log/messages: Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor
/var/log/libvirt/qemu/aries.log: 2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid 2713 (/usr/sbin/libvirtd) 2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown
============================================================================================= On the first VM effected (same thing on others): /var/log/ovirt-guest-agent/ovirt-guest-agent.log: MainThread::INFO::2020-06-25 15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent CredServer::INFO::2020-06-25 15:25:20,626::CredServer::262::root::CredServer has stopped. MainThread::INFO::2020-06-25 15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down.
============================================================================================= Packages version installated: Host OS version: CentOS 7.7.1908: ovirt-hosted-engine-ha-2.3.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-release43-4.3.6-1.el7.noarch ovirt-imageio-daemon-1.5.2-0.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch ovirt-imageio-common-1.5.2-0.el7.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 libvirt-4.5.0-23.el7_7.1.x86_64 libvirt-daemon-4.5.0-23.el7_7.1.x86_6 qemu-kvm-ev-2.12.0-33.1.el7.x86_64 qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64
On guest VM: ovirt-guest-agent-1.0.13-1.el6.noarch qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64 _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LGQSLTNG37VZDJ...
-- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev@redhat.com | lveyde@redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Hello, Thanks for your interest and time. Here is the engine log: 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 50 threads waiting for tasks and 0 tasks in queue. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 5 threads waiting for tasks. 2020-06-25 15:14:23,734-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK, 279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}' 2020-06-25 15:14:23,734-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK, 279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}' 2020-06-25 15:14:23,741-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-25 15:14:23,741-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Before acquiring lock in order to prevent monitoring for host 'athos' from data-center 'HyperVirtDC' 2020-06-25 15:14:23,741-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Lock acquired, from now a monitoring of host will be skipped for host 'athos' from data-center 'HyperVirtDC' 2020-06-25 15:14:24,574-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-2) [705ef5d8] START, GetHardwareInfoAsyncVDSCommand(HostName = athos, VdsIdAndVdsVDSCommandParametersBase:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vds='Host[athos,279ef2ed-c140-4e57-acf6-d5da176ee237]'}), log id: 67a7033d 2020-06-25 15:14:24,574-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-2) [705ef5d8] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 67a7033d 2020-06-25 15:14:24,600-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-2) [705ef5d8] EVENT_ID: VDS_NO_SELINUX_ENFORCEMENT(25), Host athos does not enforce SELinux. Current status: DISABLED 2020-06-25 15:14:24,600-05 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-2) [705ef5d8] Host 'athos' is running with SELinux in 'DISABLED' mode 2020-06-25 15:14:24,792-05 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ForkJoinPool-1-worker-2) [f683f6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS 2020-06-25 15:14:24,800-05 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ForkJoinPool-1-worker-2) [28c8cfc4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS 2020-06-25 15:14:24,804-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [28c8cfc4] Refresh host capabilities finished. Lock released. Monitoring can run now for host 'athos' from data-center 'HyperVirtDC' 2020-06-25 15:14:24,818-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-2) [28c8cfc4] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host athos. 2020-06-25 15:14:24,818-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [28c8cfc4] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK, 279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}' 2020-06-25 15:15:19,377-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default task-596) [353aaece] Running command: AddDeprecatedApiEventCommand internal: false. 2020-06-25 15:15:19,438-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [7f655795-8507-4195-8019-07721e14b519] START, GetFileStatsVDSCommand( GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109', ignoreFailoverLimit='false'}), log id: 29ea5047 2020-06-25 15:15:19,474-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [7f655795-8507-4195-8019-07721e14b519] FINISH, GetFileStatsVDSCommand, return: {virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080}, Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680}, SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0, size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0, size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992}, proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456} , oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656}, SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0, size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0, size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0, size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0, size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120}, Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432}, oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log id: 29ea5047 2020-06-25 15:15:19,529-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-596) [7f655795-8507-4195-8019-07721e14b519] EVENT_ID: REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s): HyperSense-ISO (All file type) 2020-06-25 15:20:19,374-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default task-598) [37cbaf94] Running command: AddDeprecatedApiEventCommand internal: false. 2020-06-25 15:20:19,432-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-598) [4bf1c465-8ba0-430c-951f-765502165399] START, GetFileStatsVDSCommand( GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109', ignoreFailoverLimit='false'}), log id: 57f73a38 2020-06-25 15:20:19,467-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-598) [4bf1c465-8ba0-430c-951f-765502165399] FINISH, GetFileStatsVDSCommand, return: {virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080}, Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680}, SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0, size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0, size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992}, proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456} , oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656}, SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0, size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0, size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0, size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0, size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120}, Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432}, oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log id: 57f73a38 2020-06-25 15:20:19,516-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-598) [4bf1c465-8ba0-430c-951f-765502165399] EVENT_ID: REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s): HyperSense-ISO (All file type) 2020-06-25 15:21:19,235-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS, HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]', sharedLocks=''}' 2020-06-25 15:21:19,235-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS, HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]', sharedLocks=''}' 2020-06-25 15:21:19,238-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-25 15:21:19,238-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Before acquiring lock in order to prevent monitoring for host 'goliath' from data-center 'HyperVirtDC' 2020-06-25 15:21:19,239-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Lock acquired, from now a monitoring of host will be skipped for host 'goliath' from data-center 'HyperVirtDC' 2020-06-25 15:21:21,981-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] START, GetHardwareInfoAsyncVDSCommand(HostName = goliath, VdsIdAndVdsVDSCommandParametersBase:{hostId='e3e96cd8-ac68-4317-998c-30fcdaa28571', vds='Host[goliath,e3e96cd8-ac68-4317-998c-30fcdaa28571]'}), log id: 2577d61c 2020-06-25 15:21:21,982-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 2577d61c 2020-06-25 15:21:22,000-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) [3f4ea7b6] EVENT_ID: VDS_NO_SELINUX_ENFORCEMENT(25), Host goliath does not enforce SELinux. Current status: DISABLED 2020-06-25 15:21:22,000-05 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-4) [3f4ea7b6] Host 'goliath' is running with SELinux in 'DISABLED' mode 2020-06-25 15:21:22,089-05 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ForkJoinPool-1-worker-4) [37da5bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS 2020-06-25 15:21:22,097-05 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ForkJoinPool-1-worker-4) [5ea6c697] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS 2020-06-25 15:21:22,100-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [5ea6c697] Refresh host capabilities finished. Lock released. Monitoring can run now for host 'goliath' from data-center 'HyperVirtDC' 2020-06-25 15:21:22,111-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) [5ea6c697] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host goliath. 2020-06-25 15:21:22,112-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [5ea6c697] Lock freed to object 'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS, HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]', sharedLocks=''}' 2020-06-25 15:23:56,108-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 50 threads waiting for tasks and 0 tasks in queue. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 5 threads waiting for tasks. 2020-06-25 15:25:19,376-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default task-596) [9892273] Running command: AddDeprecatedApiEventCommand internal: false. 2020-06-25 15:25:19,441-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] START, GetFileStatsVDSCommand( GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109', ignoreFailoverLimit='false'}), log id: 6be2df6 2020-06-25 15:25:19,481-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] FINISH, GetFileStatsVDSCommand, return: {virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080}, Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680}, SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0, size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0, size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992}, proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456} , oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656}, SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0, size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0, size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0, size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0, size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120}, Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432}, oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log id: 6be2df6 2020-06-25 15:25:19,504-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM 'e0257b06-28fd-4d41-83a9-adf1904d3622' was reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos) 2020-06-25 15:25:19,505-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] START, DestroyVDSCommand(HostName = athos, DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vmId='e0257b06-28fd-4d41-83a9-adf1904d3622', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 3dca5d1b 2020-06-25 15:25:19,515-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] FINISH, DestroyVDSCommand, return: , log id: 3dca5d1b 2020-06-25 15:25:19,515-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM 'e0257b06-28fd-4d41-83a9-adf1904d3622'(nyhyperprint) moved from 'Up' --> 'Down' 2020-06-25 15:25:19,530-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] EVENT_ID: REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s): HyperSense-ISO (All file type) 2020-06-25 15:25:19,610-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-6) [] EVENT_ID: VM_DOWN(61), VM nyhyperprint is down. Exit message: User shut down from within the guest 2020-06-25 15:25:19,635-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-22924) [7eaa80e3] Running command: ProcessDownVmCommand internal: true. 2020-06-25 15:25:30,772-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617' was reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos) 2020-06-25 15:25:30,773-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-15) [] START, DestroyVDSCommand(HostName = athos, DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vmId='d4418a18-46e4-445b-b344-49aa10861617', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 69aa3373 2020-06-25 15:25:30,776-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-15) [] FINISH, DestroyVDSCommand, return: , log id: 69aa3373 2020-06-25 15:25:30,776-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617'(aries) moved from 'Up' --> 'Down' 2020-06-25 15:25:30,876-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-15) [] EVENT_ID: VM_DOWN(61), VM aries is down. Exit message: User shut down from within the guest 2020-06-25 15:25:30,898-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-22900) [4e3a6bbe] Running command: ProcessDownVmCommand internal: true. 2020-06-25 15:25:31,060-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Fetched 2 VMs from VDS '279ef2ed-c140-4e57-acf6-d5da176ee237' 2020-06-25 15:25:45,266-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM '632e98d0-4262-4e20-8e31-6d94e4401193' was reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos) 2020-06-25 15:25:45,268-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] START, DestroyVDSCommand(HostName = athos, DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vmId='632e98d0-4262-4e20-8e31-6d94e4401193', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 2e0f4bde 2020-06-25 15:25:45,272-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] FINISH, DestroyVDSCommand, return: , log id: 2e0f4bde
Have you checked the qemu log on the host that was running the VMs ? What was done recently ? Any reason SELINUX is disabled ? Best Regards, Strahil Nikolov На 30 юни 2020 г. 18:08:09 GMT+03:00, Antoine Nguyen <bobbysch@gmail.com> написа:
Hello,
Thanks for your interest and time. Here is the engine log: 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 50 threads waiting for tasks and 0 tasks in queue. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks. 2020-06-25 15:13:56,105-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 5 threads waiting for tasks. 2020-06-25 15:14:23,734-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK, 279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}' 2020-06-25 15:14:23,734-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK, 279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}' 2020-06-25 15:14:23,741-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-25 15:14:23,741-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Before acquiring lock in order to prevent monitoring for host 'athos' from data-center 'HyperVirtDC' 2020-06-25 15:14:23,741-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [705ef5d8] Lock acquired, from now a monitoring of host will be skipped for host 'athos' from data-center 'HyperVirtDC' 2020-06-25 15:14:24,574-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-2) [705ef5d8] START, GetHardwareInfoAsyncVDSCommand(HostName = athos, VdsIdAndVdsVDSCommandParametersBase:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vds='Host[athos,279ef2ed-c140-4e57-acf6-d5da176ee237]'}), log id: 67a7033d 2020-06-25 15:14:24,574-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-2) [705ef5d8] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 67a7033d 2020-06-25 15:14:24,600-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-2) [705ef5d8] EVENT_ID: VDS_NO_SELINUX_ENFORCEMENT(25), Host athos does not enforce SELinux. Current status: DISABLED 2020-06-25 15:14:24,600-05 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-2) [705ef5d8] Host 'athos' is running with SELinux in 'DISABLED' mode 2020-06-25 15:14:24,792-05 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ForkJoinPool-1-worker-2) [f683f6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS 2020-06-25 15:14:24,800-05 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ForkJoinPool-1-worker-2) [28c8cfc4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS 2020-06-25 15:14:24,804-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [28c8cfc4] Refresh host capabilities finished. Lock released. Monitoring can run now for host 'athos' from data-center 'HyperVirtDC' 2020-06-25 15:14:24,818-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-2) [28c8cfc4] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host athos. 2020-06-25 15:14:24,818-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2) [28c8cfc4] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK, 279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}' 2020-06-25 15:15:19,377-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default task-596) [353aaece] Running command: AddDeprecatedApiEventCommand internal: false. 2020-06-25 15:15:19,438-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [7f655795-8507-4195-8019-07721e14b519] START, GetFileStatsVDSCommand( GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109', ignoreFailoverLimit='false'}), log id: 29ea5047 2020-06-25 15:15:19,474-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [7f655795-8507-4195-8019-07721e14b519] FINISH, GetFileStatsVDSCommand, return: {virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080}, Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680}, SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0, size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0, size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992}, proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456} , oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656}, SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0, size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0, size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0, size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0, size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120}, Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432}, oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log id: 29ea5047 2020-06-25 15:15:19,529-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-596) [7f655795-8507-4195-8019-07721e14b519] EVENT_ID: REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s): HyperSense-ISO (All file type) 2020-06-25 15:20:19,374-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default task-598) [37cbaf94] Running command: AddDeprecatedApiEventCommand internal: false. 2020-06-25 15:20:19,432-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-598) [4bf1c465-8ba0-430c-951f-765502165399] START, GetFileStatsVDSCommand( GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109', ignoreFailoverLimit='false'}), log id: 57f73a38 2020-06-25 15:20:19,467-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-598) [4bf1c465-8ba0-430c-951f-765502165399] FINISH, GetFileStatsVDSCommand, return: {virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080}, Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680}, SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0, size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0, size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992}, proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456} , oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656}, SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0, size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0, size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0, size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0, size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120}, Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432}, oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log id: 57f73a38 2020-06-25 15:20:19,516-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-598) [4bf1c465-8ba0-430c-951f-765502165399] EVENT_ID: REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s): HyperSense-ISO (All file type) 2020-06-25 15:21:19,235-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS, HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]', sharedLocks=''}' 2020-06-25 15:21:19,235-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS, HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]', sharedLocks=''}' 2020-06-25 15:21:19,238-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-25 15:21:19,238-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Before acquiring lock in order to prevent monitoring for host 'goliath' from data-center 'HyperVirtDC' 2020-06-25 15:21:19,239-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] Lock acquired, from now a monitoring of host will be skipped for host 'goliath' from data-center 'HyperVirtDC' 2020-06-25 15:21:21,981-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] START, GetHardwareInfoAsyncVDSCommand(HostName = goliath, VdsIdAndVdsVDSCommandParametersBase:{hostId='e3e96cd8-ac68-4317-998c-30fcdaa28571', vds='Host[goliath,e3e96cd8-ac68-4317-998c-30fcdaa28571]'}), log id: 2577d61c 2020-06-25 15:21:21,982-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (ForkJoinPool-1-worker-4) [3f4ea7b6] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 2577d61c 2020-06-25 15:21:22,000-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) [3f4ea7b6] EVENT_ID: VDS_NO_SELINUX_ENFORCEMENT(25), Host goliath does not enforce SELinux. Current status: DISABLED 2020-06-25 15:21:22,000-05 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-4) [3f4ea7b6] Host 'goliath' is running with SELinux in 'DISABLED' mode 2020-06-25 15:21:22,089-05 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ForkJoinPool-1-worker-4) [37da5bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS 2020-06-25 15:21:22,097-05 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ForkJoinPool-1-worker-4) [5ea6c697] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS 2020-06-25 15:21:22,100-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [5ea6c697] Refresh host capabilities finished. Lock released. Monitoring can run now for host 'goliath' from data-center 'HyperVirtDC' 2020-06-25 15:21:22,111-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) [5ea6c697] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host goliath. 2020-06-25 15:21:22,112-05 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4) [5ea6c697] Lock freed to object 'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS, HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]', sharedLocks=''}' 2020-06-25 15:23:56,108-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 50 threads waiting for tasks and 0 tasks in queue. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks. 2020-06-25 15:23:56,109-05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 5 threads waiting for tasks. 2020-06-25 15:25:19,376-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default task-596) [9892273] Running command: AddDeprecatedApiEventCommand internal: false. 2020-06-25 15:25:19,441-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] START, GetFileStatsVDSCommand( GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109', ignoreFailoverLimit='false'}), log id: 6be2df6 2020-06-25 15:25:19,481-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] FINISH, GetFileStatsVDSCommand, return: {virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080}, Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680}, SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0, size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0, size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992}, proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456} , oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656}, SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0, size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0, size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0, size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0, size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120}, Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432}, oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log id: 6be2df6 2020-06-25 15:25:19,504-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM 'e0257b06-28fd-4d41-83a9-adf1904d3622' was reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos) 2020-06-25 15:25:19,505-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] START, DestroyVDSCommand(HostName = athos, DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vmId='e0257b06-28fd-4d41-83a9-adf1904d3622', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 3dca5d1b 2020-06-25 15:25:19,515-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] FINISH, DestroyVDSCommand, return: , log id: 3dca5d1b 2020-06-25 15:25:19,515-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM 'e0257b06-28fd-4d41-83a9-adf1904d3622'(nyhyperprint) moved from 'Up' --> 'Down' 2020-06-25 15:25:19,530-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] EVENT_ID: REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s): HyperSense-ISO (All file type) 2020-06-25 15:25:19,610-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-6) [] EVENT_ID: VM_DOWN(61), VM nyhyperprint is down. Exit message: User shut down from within the guest 2020-06-25 15:25:19,635-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-22924) [7eaa80e3] Running command: ProcessDownVmCommand internal: true. 2020-06-25 15:25:30,772-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617' was reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos) 2020-06-25 15:25:30,773-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-15) [] START, DestroyVDSCommand(HostName = athos, DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vmId='d4418a18-46e4-445b-b344-49aa10861617', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 69aa3373 2020-06-25 15:25:30,776-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-15) [] FINISH, DestroyVDSCommand, return: , log id: 69aa3373 2020-06-25 15:25:30,776-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617'(aries) moved from 'Up' --> 'Down' 2020-06-25 15:25:30,876-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-15) [] EVENT_ID: VM_DOWN(61), VM aries is down. Exit message: User shut down from within the guest 2020-06-25 15:25:30,898-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-22900) [4e3a6bbe] Running command: ProcessDownVmCommand internal: true. 2020-06-25 15:25:31,060-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Fetched 2 VMs from VDS '279ef2ed-c140-4e57-acf6-d5da176ee237' 2020-06-25 15:25:45,266-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM '632e98d0-4262-4e20-8e31-6d94e4401193' was reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos) 2020-06-25 15:25:45,268-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] START, DestroyVDSCommand(HostName = athos, DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237', vmId='632e98d0-4262-4e20-8e31-6d94e4401193', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 2e0f4bde 2020-06-25 15:25:45,272-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) [] FINISH, DestroyVDSCommand, return: , log id: 2e0f4bde _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SDUWOYANYZ27RQ...
Here is what I found in the audit log: on Centos 7: type=SERVICE_START msg=audit(06/25/2020 15:24:18.967:80755) : pid=1 uid=root auid=unset ses=unset msg='unit=systemd-shutdownd comm=systemd exe=/usr/lib/systemd/systemd hostname=? addr=? terminal=? res=success' type=SYSTEM_RUNLEVEL msg=audit(06/25/2020 15:25:19.501:103256) : user pid=12951 uid=root auid=unset ses=unset msg='old-level=3 new-level=0 exe=/sbin/shutdown hostname=? addr=? terminal=? res=success' ---- type=SYSTEM_SHUTDOWN msg=audit(06/25/2020 15:25:19.501:103257) : user pid=12951 uid=root auid=unset ses=unset msg='init exe=/sbin/shutdown hostname=? addr=? terminal=? res=success' ============================================================= on Centos 6: [root@aries ~]# ausearch -i -p 12951 ---- type=USER_AUTH msg=audit(05/23/2020 20:23:44.890:8878) : user pid=12951 uid=ovirtagent auid=unset ses=unset msg='op=PAM:authentication acct=ovirtagent exe=/usr/sbin/userhelper hostname=? addr=? terminal=? res=success' ---- type=USER_ACCT msg=audit(05/23/2020 20:23:44.891:8879) : user pid=12951 uid=ovirtagent auid=unset ses=unset msg='op=PAM:accounting acct=ovirtagent exe=/usr/sbin/userhelper hostname=? addr=? terminal=? res=success' ---- type=SYSTEM_RUNLEVEL msg=audit(06/25/2020 15:25:19.501:103256) : user pid=12951 uid=root auid=unset ses=unset msg='old-level=3 new-level=0 exe=/sbin/shutdown hostname=? addr=? terminal=? res=success' ---- type=SYSTEM_SHUTDOWN msg=audit(06/25/2020 15:25:19.501:103257) : user pid=12951 uid=root auid=unset ses=unset msg='init exe=/sbin/shutdown hostname=? addr=? terminal=? res=success' ----
participants (4)
-
Antoine Nguyen -
Bobby
-
Lev Veyde -
Strahil Nikolov