10:37 p.m.
Hello,
All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason
almost simultaneously.
Please help me to find the root cause.
Thanks.
Please note the host seems doing fine and never crash or hangs and I can
migrate VMs back to it later.
Here is the exact timeline of all the related events combined from the host
and the VM(s):
On oVirt host:
/var/log/vdsm/vdsm.log:
2020-06-25 15:25:16,944-0500 WARN (qgapoller/3)
[virt.periodic.VmDispatcher] could not run <function <lambda> at
0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289)
2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File:
/var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0
already removed (fileutils:54)
2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File:
/var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0
already removed (fileutils:54)
[root@athos log]# journalctl -u NetworkManager --since=today
-- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25
16:36:05 CDT. --
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136]
device (vnet0): state change: disconnected -> unmanaged (reason
'unmanaged', sys-iface-state: 'removed')
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146]
device (vnet0): released from master device SRV-VL
/var/log/messages:
Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136]
device (vnet0): state change: disconnected -> unmanaged (reason
'unmanaged', sys-iface-state: 'removed')
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146]
device (vnet0): released from master device SRV-VL
Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error :
qemuMonitorIO:718 : internal error: End of file from qemu monitor
/var/log/libvirt/qemu/aries.log:
2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid
2713 (/usr/sbin/libvirtd)
2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown
=============================================================================================
On the first VM effected (same thing on others):
/var/log/ovirt-guest-agent/ovirt-guest-agent.log:
MainThread::INFO::2020-06-25
15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent
CredServer::INFO::2020-06-25
15:25:20,626::CredServer::262::root::CredServer has stopped.
MainThread::INFO::2020-06-25
15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down.
=============================================================================================
Packages version installated:
Host OS version: CentOS 7.7.1908:
ovirt-hosted-engine-ha-2.3.5-1.el7.noarch
ovirt-provider-ovn-driver-1.2.22-1.el7.noarch
ovirt-release43-4.3.6-1.el7.noarch
ovirt-imageio-daemon-1.5.2-0.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-imageio-common-1.5.2-0.el7.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
ovirt-vmconsole-host-1.0.7-2.el7.noarch
ovirt-host-4.3.4-1.el7.x86_64
libvirt-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-4.5.0-23.el7_7.1.x86_6
qemu-kvm-ev-2.12.0-33.1.el7.x86_64
qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64
On guest VM:
ovirt-guest-agent-1.0.13-1.el6.noarch
qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64
1:09 a.m.
Hi Bobby,
Can you please share the engine logs as well?
It could help to understand what happened there.
Right now, looking at the pieces of the logs you sent I couldn't spot
anything unusual.
Thanks in advance,
On Mon, Jun 29, 2020 at 10:40 PM Bobby <bobbysch(a)gmail.com> wrote:
Hello,
All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason
almost simultaneously.
Please help me to find the root cause.
Thanks.
Please note the host seems doing fine and never crash or hangs and I can
migrate VMs back to it later.
Here is the exact timeline of all the related events combined from the
host and the VM(s):
On oVirt host:
/var/log/vdsm/vdsm.log:
2020-06-25 15:25:16,944-0500 WARN (qgapoller/3)
[virt.periodic.VmDispatcher] could not run <function <lambda> at
0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289)
2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File:
/var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0
already removed (fileutils:54)
2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File:
/var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0
already removed (fileutils:54)
[root@athos log]# journalctl -u NetworkManager --since=today
-- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25
16:36:05 CDT. --
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136]
device (vnet0): state change: disconnected -> unmanaged (reason
'unmanaged', sys-iface-state: 'removed')
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146]
device (vnet0): released from master device SRV-VL
/var/log/messages:
Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136]
device (vnet0): state change: disconnected -> unmanaged (reason
'unmanaged', sys-iface-state: 'removed')
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146]
device (vnet0): released from master device SRV-VL
Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error
: qemuMonitorIO:718 : internal error: End of file from qemu monitor
/var/log/libvirt/qemu/aries.log:
2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid
2713 (/usr/sbin/libvirtd)
2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown
=============================================================================================
On the first VM effected (same thing on others):
/var/log/ovirt-guest-agent/ovirt-guest-agent.log:
MainThread::INFO::2020-06-25
15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent
CredServer::INFO::2020-06-25
15:25:20,626::CredServer::262::root::CredServer has stopped.
MainThread::INFO::2020-06-25
15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down.
=============================================================================================
Packages version installated:
Host OS version: CentOS 7.7.1908:
ovirt-hosted-engine-ha-2.3.5-1.el7.noarch
ovirt-provider-ovn-driver-1.2.22-1.el7.noarch
ovirt-release43-4.3.6-1.el7.noarch
ovirt-imageio-daemon-1.5.2-0.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-imageio-common-1.5.2-0.el7.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
ovirt-vmconsole-host-1.0.7-2.el7.noarch
ovirt-host-4.3.4-1.el7.x86_64
libvirt-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-4.5.0-23.el7_7.1.x86_6
qemu-kvm-ev-2.12.0-33.1.el7.x86_64
qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64
On guest VM:
ovirt-guest-agent-1.0.13-1.el6.noarch
qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LGQSLTNG37V...
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
lev(a)redhat.com | lveyde(a)redhat.com
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
6:08 p.m.
Hello,
Thanks for your interest and time.
Here is the engine log:
2020-06-25 15:13:56,105-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default'
is using 0 threads out of 1, 5 threads waiting for tasks.
2020-06-25 15:13:56,105-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine'
is using 0 threads out of 500, 50 threads waiting for tasks and 0 tasks in queue.
2020-06-25 15:13:56,105-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks.
2020-06-25 15:13:56,105-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for
tasks.
2020-06-25 15:13:56,105-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'hostUpdatesChecker' is using 0 threads out of 5, 5 threads waiting for tasks.
2020-06-25 15:14:23,734-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[705ef5d8] Before acquiring and wait lock
'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK,
279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}'
2020-06-25 15:14:23,734-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[705ef5d8] Lock-wait acquired to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK,
279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}'
2020-06-25 15:14:23,741-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[705ef5d8] Running command: RefreshHostCapabilitiesCommand internal: true. Entities
affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDSAction group MANIPULATE_HOST
with role type ADMIN
2020-06-25 15:14:23,741-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[705ef5d8] Before acquiring lock in order to prevent monitoring for host 'athos'
from data-center 'HyperVirtDC'
2020-06-25 15:14:23,741-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[705ef5d8] Lock acquired, from now a monitoring of host will be skipped for host
'athos' from data-center 'HyperVirtDC'
2020-06-25 15:14:24,574-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(ForkJoinPool-1-worker-2) [705ef5d8] START, GetHardwareInfoAsyncVDSCommand(HostName =
athos,
VdsIdAndVdsVDSCommandParametersBase:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
vds='Host[athos,279ef2ed-c140-4e57-acf6-d5da176ee237]'}), log id: 67a7033d
2020-06-25 15:14:24,574-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(ForkJoinPool-1-worker-2) [705ef5d8] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log
id: 67a7033d
2020-06-25 15:14:24,600-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-2) [705ef5d8] EVENT_ID: VDS_NO_SELINUX_ENFORCEMENT(25), Host athos
does not enforce SELinux. Current status: DISABLED
2020-06-25 15:14:24,600-05 WARN [org.ovirt.engine.core.vdsbroker.VdsManager]
(ForkJoinPool-1-worker-2) [705ef5d8] Host 'athos' is running with SELinux in
'DISABLED' mode
2020-06-25 15:14:24,792-05 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(ForkJoinPool-1-worker-2) [f683f6a] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS
2020-06-25 15:14:24,800-05 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(ForkJoinPool-1-worker-2) [28c8cfc4] Running command: HandleVdsVersionCommand internal:
true. Entities affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS
2020-06-25 15:14:24,804-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[28c8cfc4] Refresh host capabilities finished. Lock released. Monitoring can run now for
host 'athos' from data-center 'HyperVirtDC'
2020-06-25 15:14:24,818-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-2) [28c8cfc4] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606),
Successfully refreshed the capabilities of host athos.
2020-06-25 15:14:24,818-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-2)
[28c8cfc4] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK,
279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}'
2020-06-25 15:15:19,377-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand]
(default task-596) [353aaece] Running command: AddDeprecatedApiEventCommand internal:
false.
2020-06-25 15:15:19,438-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596)
[7f655795-8507-4195-8019-07721e14b519] START, GetFileStatsVDSCommand(
GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109',
ignoreFailoverLimit='false'}), log id: 29ea5047
2020-06-25 15:15:19,474-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596)
[7f655795-8507-4195-8019-07721e14b519] FINISH, GetFileStatsVDSCommand, return:
{virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384},
ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520},
oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080},
Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0,
size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680},
SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0,
size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0,
size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0,
size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992},
proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456}
, oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656},
SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0,
size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0,
size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0,
size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0,
size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120},
Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432},
oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784},
clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log
id: 29ea5047
2020-06-25 15:15:19,529-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-596)
[7f655795-8507-4195-8019-07721e14b519] EVENT_ID:
REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s):
HyperSense-ISO (All file type)
2020-06-25 15:20:19,374-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand]
(default task-598) [37cbaf94] Running command: AddDeprecatedApiEventCommand internal:
false.
2020-06-25 15:20:19,432-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-598)
[4bf1c465-8ba0-430c-951f-765502165399] START, GetFileStatsVDSCommand(
GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109',
ignoreFailoverLimit='false'}), log id: 57f73a38
2020-06-25 15:20:19,467-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-598)
[4bf1c465-8ba0-430c-951f-765502165399] FINISH, GetFileStatsVDSCommand, return:
{virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384},
ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520},
oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080},
Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0,
size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680},
SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0,
size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0,
size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0,
size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992},
proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456}
, oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656},
SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0,
size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0,
size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0,
size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0,
size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120},
Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432},
oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784},
clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log
id: 57f73a38
2020-06-25 15:20:19,516-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-598)
[4bf1c465-8ba0-430c-951f-765502165399] EVENT_ID:
REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s):
HyperSense-ISO (All file type)
2020-06-25 15:21:19,235-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[3f4ea7b6] Before acquiring and wait lock
'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS,
HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]',
sharedLocks=''}'
2020-06-25 15:21:19,235-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[3f4ea7b6] Lock-wait acquired to object
'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS,
HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]',
sharedLocks=''}'
2020-06-25 15:21:19,238-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[3f4ea7b6] Running command: RefreshHostCapabilitiesCommand internal: true. Entities
affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDSAction group MANIPULATE_HOST
with role type ADMIN
2020-06-25 15:21:19,238-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[3f4ea7b6] Before acquiring lock in order to prevent monitoring for host 'goliath'
from data-center 'HyperVirtDC'
2020-06-25 15:21:19,239-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[3f4ea7b6] Lock acquired, from now a monitoring of host will be skipped for host
'goliath' from data-center 'HyperVirtDC'
2020-06-25 15:21:21,981-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(ForkJoinPool-1-worker-4) [3f4ea7b6] START, GetHardwareInfoAsyncVDSCommand(HostName =
goliath,
VdsIdAndVdsVDSCommandParametersBase:{hostId='e3e96cd8-ac68-4317-998c-30fcdaa28571',
vds='Host[goliath,e3e96cd8-ac68-4317-998c-30fcdaa28571]'}), log id: 2577d61c
2020-06-25 15:21:21,982-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(ForkJoinPool-1-worker-4) [3f4ea7b6] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log
id: 2577d61c
2020-06-25 15:21:22,000-05 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-4) [3f4ea7b6] EVENT_ID: VDS_NO_SELINUX_ENFORCEMENT(25), Host
goliath does not enforce SELinux. Current status: DISABLED
2020-06-25 15:21:22,000-05 WARN [org.ovirt.engine.core.vdsbroker.VdsManager]
(ForkJoinPool-1-worker-4) [3f4ea7b6] Host 'goliath' is running with SELinux in
'DISABLED' mode
2020-06-25 15:21:22,089-05 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(ForkJoinPool-1-worker-4) [37da5bf] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS
2020-06-25 15:21:22,097-05 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(ForkJoinPool-1-worker-4) [5ea6c697] Running command: HandleVdsVersionCommand internal:
true. Entities affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS
2020-06-25 15:21:22,100-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[5ea6c697] Refresh host capabilities finished. Lock released. Monitoring can run now for
host 'goliath' from data-center 'HyperVirtDC'
2020-06-25 15:21:22,111-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-4) [5ea6c697] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606),
Successfully refreshed the capabilities of host goliath.
2020-06-25 15:21:22,112-05 INFO
[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-4)
[5ea6c697] Lock freed to object
'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS,
HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]',
sharedLocks=''}'
2020-06-25 15:23:56,108-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default'
is using 0 threads out of 1, 5 threads waiting for tasks.
2020-06-25 15:23:56,109-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine'
is using 0 threads out of 500, 50 threads waiting for tasks and 0 tasks in queue.
2020-06-25 15:23:56,109-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks.
2020-06-25 15:23:56,109-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for
tasks.
2020-06-25 15:23:56,109-05 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'hostUpdatesChecker' is using 0 threads out of 5, 5 threads waiting for tasks.
2020-06-25 15:25:19,376-05 INFO [org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand]
(default task-596) [9892273] Running command: AddDeprecatedApiEventCommand internal:
false.
2020-06-25 15:25:19,441-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596)
[73ae851b-2540-4874-9a64-4367dc4c738a] START, GetFileStatsVDSCommand(
GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109',
ignoreFailoverLimit='false'}), log id: 6be2df6
2020-06-25 15:25:19,481-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-596)
[73ae851b-2540-4874-9a64-4367dc4c738a] FINISH, GetFileStatsVDSCommand, return:
{virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0, size=350224384},
ovirt-tools-setup.iso={status=0, ctime=1539826781.0, size=359147520},
oVirt-toolsSetup-4.2-4.el7.iso={status=0, ctime=1544116372.0, size=363438080},
Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0, ctime=1470087436.0,
size=3696842752}, Win7Pro.iso={status=0, ctime=1470087472.0, size=3320903680},
SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0, ctime=1470087896.0,
size=796663808}, CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0,
size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0, ctime=1470087358.0,
size=358959104}, virtio-win.iso={status=0, ctime=1508782403.0, size=316628992},
proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0, size=534931456}
, oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0, size=107974656},
SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0, ctime=1470087907.0,
size=851998720}, oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0,
size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0, ctime=1508785246.0,
size=63229952}, Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0,
size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0, size=2949120},
Win10_1709_English_x64.iso={status=0, ctime=1519060396.0, size=4697362432},
oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0, size=99446784},
clonezilla-live-2.5.0-25-amd64.iso={status=0, ctime=1489676885.0, size=203423744}}, log
id: 6be2df6
2020-06-25 15:25:19,504-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-6) [] VM 'e0257b06-28fd-4d41-83a9-adf1904d3622' was
reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos)
2020-06-25 15:25:19,505-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) []
START, DestroyVDSCommand(HostName = athos,
DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
vmId='e0257b06-28fd-4d41-83a9-adf1904d3622', secondsToWait='0',
gracefully='false', reason='', ignoreNoVm='true'}), log id:
3dca5d1b
2020-06-25 15:25:19,515-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) []
FINISH, DestroyVDSCommand, return: , log id: 3dca5d1b
2020-06-25 15:25:19,515-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-6) [] VM
'e0257b06-28fd-4d41-83a9-adf1904d3622'(nyhyperprint) moved from 'Up'
--> 'Down'
2020-06-25 15:25:19,530-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-596)
[73ae851b-2540-4874-9a64-4367dc4c738a] EVENT_ID:
REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list succeeded for domain(s):
HyperSense-ISO (All file type)
2020-06-25 15:25:19,610-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-6) [] EVENT_ID: VM_DOWN(61), VM nyhyperprint is down. Exit message:
User shut down from within the guest
2020-06-25 15:25:19,635-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand]
(EE-ManagedThreadFactory-engine-Thread-22924) [7eaa80e3] Running command:
ProcessDownVmCommand internal: true.
2020-06-25 15:25:30,772-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617' was
reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos)
2020-06-25 15:25:30,773-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-15)
[] START, DestroyVDSCommand(HostName = athos,
DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
vmId='d4418a18-46e4-445b-b344-49aa10861617', secondsToWait='0',
gracefully='false', reason='', ignoreNoVm='true'}), log id:
69aa3373
2020-06-25 15:25:30,776-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-15)
[] FINISH, DestroyVDSCommand, return: , log id: 69aa3373
2020-06-25 15:25:30,776-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617'(aries)
moved from 'Up' --> 'Down'
2020-06-25 15:25:30,876-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-15) [] EVENT_ID: VM_DOWN(61), VM aries is down. Exit message: User
shut down from within the guest
2020-06-25 15:25:30,898-05 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand]
(EE-ManagedThreadFactory-engine-Thread-22900) [4e3a6bbe] Running command:
ProcessDownVmCommand internal: true.
2020-06-25 15:25:31,060-05 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
(EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Fetched 2 VMs from VDS
'279ef2ed-c140-4e57-acf6-d5da176ee237'
2020-06-25 15:25:45,266-05 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-6) [] VM '632e98d0-4262-4e20-8e31-6d94e4401193' was
reported as Down on VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'(athos)
2020-06-25 15:25:45,268-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) []
START, DestroyVDSCommand(HostName = athos,
DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
vmId='632e98d0-4262-4e20-8e31-6d94e4401193', secondsToWait='0',
gracefully='false', reason='', ignoreNoVm='true'}), log id:
2e0f4bde
2020-06-25 15:25:45,272-05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-6) []
FINISH, DestroyVDSCommand, return: , log id: 2e0f4bde
8:42 p.m.
Have you checked the qemu log on the host that was running the VMs ?
What was done recently ?
Any reason SELINUX is disabled ?
Best Regards,
Strahil Nikolov
На 30 юни 2020 г. 18:08:09 GMT+03:00, Antoine Nguyen <bobbysch(a)gmail.com> написа:
>Hello,
>
>Thanks for your interest and time.
>Here is the engine log:
>2020-06-25 15:13:56,105-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'default' is using 0 threads out of 1, 5 threads waiting for
>tasks.
>2020-06-25 15:13:56,105-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'engine' is using 0 threads out of 500, 50 threads waiting for
>tasks and 0 tasks in queue.
>2020-06-25 15:13:56,105-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'engineScheduled' is using 0 threads out of 100, 100 threads
>waiting for tasks.
>2020-06-25 15:13:56,105-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads
>waiting for tasks.
>2020-06-25 15:13:56,105-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'hostUpdatesChecker' is using 0 threads out of 5, 5 threads
>waiting for tasks.
>2020-06-25 15:14:23,734-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] Before acquiring and wait lock
>'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK,
>279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}'
>2020-06-25 15:14:23,734-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] Lock-wait acquired to object
>'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK,
>279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}'
>2020-06-25 15:14:23,741-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] Running command:
>RefreshHostCapabilitiesCommand internal: true. Entities affected : ID:
>279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDSAction group
>MANIPULATE_HOST with role type ADMIN
>2020-06-25 15:14:23,741-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] Before acquiring lock in order to
>prevent monitoring for host 'athos' from data-center 'HyperVirtDC'
>2020-06-25 15:14:23,741-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] Lock acquired, from now a
>monitoring of host will be skipped for host 'athos' from data-center
>'HyperVirtDC'
>2020-06-25 15:14:24,574-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] START,
>GetHardwareInfoAsyncVDSCommand(HostName = athos,
>VdsIdAndVdsVDSCommandParametersBase:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
>vds='Host[athos,279ef2ed-c140-4e57-acf6-d5da176ee237]'}), log id:
>67a7033d
>2020-06-25 15:14:24,574-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
>(ForkJoinPool-1-worker-2) [705ef5d8] FINISH,
>GetHardwareInfoAsyncVDSCommand, return: , log id: 67a7033d
>2020-06-25 15:14:24,600-05 WARN
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(ForkJoinPool-1-worker-2) [705ef5d8] EVENT_ID:
>VDS_NO_SELINUX_ENFORCEMENT(25), Host athos does not enforce SELinux.
>Current status: DISABLED
>2020-06-25 15:14:24,600-05 WARN
>[org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-2)
>[705ef5d8] Host 'athos' is running with SELinux in 'DISABLED' mode
>2020-06-25 15:14:24,792-05 INFO
>[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
>(ForkJoinPool-1-worker-2) [f683f6a] Running command:
>HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities
>affected : ID: 279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS
>2020-06-25 15:14:24,800-05 INFO
>[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
>(ForkJoinPool-1-worker-2) [28c8cfc4] Running command:
>HandleVdsVersionCommand internal: true. Entities affected : ID:
>279ef2ed-c140-4e57-acf6-d5da176ee237 Type: VDS
>2020-06-25 15:14:24,804-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [28c8cfc4] Refresh host capabilities
>finished. Lock released. Monitoring can run now for host 'athos' from
>data-center 'HyperVirtDC'
>2020-06-25 15:14:24,818-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(ForkJoinPool-1-worker-2) [28c8cfc4] EVENT_ID:
>HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the
>capabilities of host athos.
>2020-06-25 15:14:24,818-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-2) [28c8cfc4] Lock freed to object
>'EngineLock:{exclusiveLocks='[HOST_NETWORK279ef2ed-c140-4e57-acf6-d5da176ee237=HOST_NETWORK,
>279ef2ed-c140-4e57-acf6-d5da176ee237=VDS]', sharedLocks=''}'
>2020-06-25 15:15:19,377-05 INFO
>[org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default
>task-596) [353aaece] Running command: AddDeprecatedApiEventCommand
>internal: false.
>2020-06-25 15:15:19,438-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
>(default task-596) [7f655795-8507-4195-8019-07721e14b519] START,
>GetFileStatsVDSCommand(
>GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109',
>ignoreFailoverLimit='false'}), log id: 29ea5047
>2020-06-25 15:15:19,474-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
>(default task-596) [7f655795-8507-4195-8019-07721e14b519] FINISH,
>GetFileStatsVDSCommand, return:
>{virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0,
>size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0,
>size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0,
>ctime=1544116372.0, size=363438080},
>Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0,
>ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0,
>ctime=1470087472.0, size=3320903680},
>SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0,
>ctime=1470087896.0, size=796663808},
>CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0,
>size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0,
>ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0,
>ctime=1508782403.0, size=316628992},
>proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0,
>size=534931456}
>, oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0,
>size=107974656},
>SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0,
>ctime=1470087907.0, size=851998720},
>oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0,
>size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0,
>ctime=1508785246.0, size=63229952},
>Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0,
>size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0,
>size=2949120}, Win10_1709_English_x64.iso={status=0,
>ctime=1519060396.0, size=4697362432},
>oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0,
>size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0,
>ctime=1489676885.0, size=203423744}}, log id: 29ea5047
>2020-06-25 15:15:19,529-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(default task-596) [7f655795-8507-4195-8019-07721e14b519] EVENT_ID:
>REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list
>succeeded for domain(s): HyperSense-ISO (All file type)
>2020-06-25 15:20:19,374-05 INFO
>[org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default
>task-598) [37cbaf94] Running command: AddDeprecatedApiEventCommand
>internal: false.
>2020-06-25 15:20:19,432-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
>(default task-598) [4bf1c465-8ba0-430c-951f-765502165399] START,
>GetFileStatsVDSCommand(
>GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109',
>ignoreFailoverLimit='false'}), log id: 57f73a38
>2020-06-25 15:20:19,467-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
>(default task-598) [4bf1c465-8ba0-430c-951f-765502165399] FINISH,
>GetFileStatsVDSCommand, return:
>{virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0,
>size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0,
>size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0,
>ctime=1544116372.0, size=363438080},
>Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0,
>ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0,
>ctime=1470087472.0, size=3320903680},
>SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0,
>ctime=1470087896.0, size=796663808},
>CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0,
>size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0,
>ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0,
>ctime=1508782403.0, size=316628992},
>proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0,
>size=534931456}
>, oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0,
>size=107974656},
>SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0,
>ctime=1470087907.0, size=851998720},
>oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0,
>size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0,
>ctime=1508785246.0, size=63229952},
>Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0,
>size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0,
>size=2949120}, Win10_1709_English_x64.iso={status=0,
>ctime=1519060396.0, size=4697362432},
>oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0,
>size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0,
>ctime=1489676885.0, size=203423744}}, log id: 57f73a38
>2020-06-25 15:20:19,516-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(default task-598) [4bf1c465-8ba0-430c-951f-765502165399] EVENT_ID:
>REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list
>succeeded for domain(s): HyperSense-ISO (All file type)
>2020-06-25 15:21:19,235-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] Before acquiring and wait lock
>'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS,
>HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]',
>sharedLocks=''}'
>2020-06-25 15:21:19,235-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] Lock-wait acquired to object
>'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS,
>HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]',
>sharedLocks=''}'
>2020-06-25 15:21:19,238-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] Running command:
>RefreshHostCapabilitiesCommand internal: true. Entities affected : ID:
>e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDSAction group
>MANIPULATE_HOST with role type ADMIN
>2020-06-25 15:21:19,238-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] Before acquiring lock in order to
>prevent monitoring for host 'goliath' from data-center 'HyperVirtDC'
>2020-06-25 15:21:19,239-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] Lock acquired, from now a
>monitoring of host will be skipped for host 'goliath' from data-center
>'HyperVirtDC'
>2020-06-25 15:21:21,981-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] START,
>GetHardwareInfoAsyncVDSCommand(HostName = goliath,
>VdsIdAndVdsVDSCommandParametersBase:{hostId='e3e96cd8-ac68-4317-998c-30fcdaa28571',
>vds='Host[goliath,e3e96cd8-ac68-4317-998c-30fcdaa28571]'}), log id:
>2577d61c
>2020-06-25 15:21:21,982-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] FINISH,
>GetHardwareInfoAsyncVDSCommand, return: , log id: 2577d61c
>2020-06-25 15:21:22,000-05 WARN
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(ForkJoinPool-1-worker-4) [3f4ea7b6] EVENT_ID:
>VDS_NO_SELINUX_ENFORCEMENT(25), Host goliath does not enforce SELinux.
>Current status: DISABLED
>2020-06-25 15:21:22,000-05 WARN
>[org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-4)
>[3f4ea7b6] Host 'goliath' is running with SELinux in 'DISABLED' mode
>2020-06-25 15:21:22,089-05 INFO
>[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
>(ForkJoinPool-1-worker-4) [37da5bf] Running command:
>HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities
>affected : ID: e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS
>2020-06-25 15:21:22,097-05 INFO
>[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
>(ForkJoinPool-1-worker-4) [5ea6c697] Running command:
>HandleVdsVersionCommand internal: true. Entities affected : ID:
>e3e96cd8-ac68-4317-998c-30fcdaa28571 Type: VDS
>2020-06-25 15:21:22,100-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [5ea6c697] Refresh host capabilities
>finished. Lock released. Monitoring can run now for host 'goliath' from
>data-center 'HyperVirtDC'
>2020-06-25 15:21:22,111-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(ForkJoinPool-1-worker-4) [5ea6c697] EVENT_ID:
>HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the
>capabilities of host goliath.
>2020-06-25 15:21:22,112-05 INFO
>[org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand]
>(ForkJoinPool-1-worker-4) [5ea6c697] Lock freed to object
>'EngineLock:{exclusiveLocks='[e3e96cd8-ac68-4317-998c-30fcdaa28571=VDS,
>HOST_NETWORKe3e96cd8-ac68-4317-998c-30fcdaa28571=HOST_NETWORK]',
>sharedLocks=''}'
>2020-06-25 15:23:56,108-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'default' is using 0 threads out of 1, 5 threads waiting for
>tasks.
>2020-06-25 15:23:56,109-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'engine' is using 0 threads out of 500, 50 threads waiting for
>tasks and 0 tasks in queue.
>2020-06-25 15:23:56,109-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'engineScheduled' is using 0 threads out of 100, 100 threads
>waiting for tasks.
>2020-06-25 15:23:56,109-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads
>waiting for tasks.
>2020-06-25 15:23:56,109-05 INFO
>[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
>(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread
>pool 'hostUpdatesChecker' is using 0 threads out of 5, 5 threads
>waiting for tasks.
>2020-06-25 15:25:19,376-05 INFO
>[org.ovirt.engine.core.bll.AddDeprecatedApiEventCommand] (default
>task-596) [9892273] Running command: AddDeprecatedApiEventCommand
>internal: false.
>2020-06-25 15:25:19,441-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
>(default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] START,
>GetFileStatsVDSCommand(
>GetFileStatsParameters:{storagePoolId='6fbfe101-a34d-4cb9-a4c6-b205f534a109',
>ignoreFailoverLimit='false'}), log id: 6be2df6
>2020-06-25 15:25:19,481-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand]
>(default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] FINISH,
>GetFileStatsVDSCommand, return:
>{virt-p2v-livecd-1.28.1-1.49.1.el7.iso={status=0, ctime=1469478434.0,
>size=350224384}, ovirt-tools-setup.iso={status=0, ctime=1539826781.0,
>size=359147520}, oVirt-toolsSetup-4.2-4.el7.iso={status=0,
>ctime=1544116372.0, size=363438080},
>Win10.TH1_CLIENTENTERPRISE_S_EVAL_X64FRE_EN-US.ISO={status=0,
>ctime=1470087436.0, size=3696842752}, Win7Pro.iso={status=0,
>ctime=1470087472.0, size=3320903680},
>SW_DVD5_Office_2013w_SP1_64Bit_English_MLF_X19-34904.iso={status=0,
>ctime=1470087896.0, size=796663808},
>CentOS-7-x86_64-Minimal-1511.iso={status=0, ctime=1470087365.0,
>size=632291328}, CentOS-6.4-x86_64-minimal.iso={status=0,
>ctime=1470087358.0, size=358959104}, virtio-win.iso={status=0,
>ctime=1508782403.0, size=316628992},
>proxmox-ve_4.3-e7cdc165-2.iso={status=0, ctime=1478538195.0,
>size=534931456}
>, oVirt-toolsSetup_3.5_7.iso={status=0, ctime=1489522124.0,
>size=107974656},
>SW_DVD5_Office_2016_64Bit_English_MLF_X20-42479.iso={status=0,
>ctime=1470087907.0, size=851998720},
>oVirt-toolsSetup_4.1-3.fc24.iso={status=0, ctime=1508471232.0,
>size=220481536}, flexvdi-guest-tools-3.0.0.iso={status=0,
>ctime=1508785246.0, size=63229952},
>Fedora-Server-dvd-x86_64-30-1.2.iso={status=0, ctime=1568390493.0,
>size=3177185280}, virtio-win_amd64.vfd={status=0, ctime=1508782419.0,
>size=2949120}, Win10_1709_English_x64.iso={status=0,
>ctime=1519060396.0, size=4697362432},
>oVirt-toolsSetup_4.0-1.fc23.iso={status=0, ctime=1468957441.0,
>size=99446784}, clonezilla-live-2.5.0-25-amd64.iso={status=0,
>ctime=1489676885.0, size=203423744}}, log id: 6be2df6
>2020-06-25 15:25:19,504-05 INFO
>[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>(ForkJoinPool-1-worker-6) [] VM 'e0257b06-28fd-4d41-83a9-adf1904d3622'
>was reported as Down on VDS
>'279ef2ed-c140-4e57-acf6-d5da176ee237'(athos)
>2020-06-25 15:25:19,505-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
>(ForkJoinPool-1-worker-6) [] START, DestroyVDSCommand(HostName = athos,
>DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
>vmId='e0257b06-28fd-4d41-83a9-adf1904d3622', secondsToWait='0',
>gracefully='false', reason='', ignoreNoVm='true'}), log id:
3dca5d1b
>2020-06-25 15:25:19,515-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
>(ForkJoinPool-1-worker-6) [] FINISH, DestroyVDSCommand, return: , log
>id: 3dca5d1b
>2020-06-25 15:25:19,515-05 INFO
>[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>(ForkJoinPool-1-worker-6) [] VM
>'e0257b06-28fd-4d41-83a9-adf1904d3622'(nyhyperprint) moved from 'Up'
>--> 'Down'
>2020-06-25 15:25:19,530-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(default task-596) [73ae851b-2540-4874-9a64-4367dc4c738a] EVENT_ID:
>REFRESH_REPOSITORY_IMAGE_LIST_SUCCEEDED(998), Refresh image list
>succeeded for domain(s): HyperSense-ISO (All file type)
>2020-06-25 15:25:19,610-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(ForkJoinPool-1-worker-6) [] EVENT_ID: VM_DOWN(61), VM nyhyperprint is
>down. Exit message: User shut down from within the guest
>2020-06-25 15:25:19,635-05 INFO
>[org.ovirt.engine.core.bll.ProcessDownVmCommand]
>(EE-ManagedThreadFactory-engine-Thread-22924) [7eaa80e3] Running
>command: ProcessDownVmCommand internal: true.
>2020-06-25 15:25:30,772-05 INFO
>[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>(ForkJoinPool-1-worker-15) [] VM 'd4418a18-46e4-445b-b344-49aa10861617'
>was reported as Down on VDS
>'279ef2ed-c140-4e57-acf6-d5da176ee237'(athos)
>2020-06-25 15:25:30,773-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
>(ForkJoinPool-1-worker-15) [] START, DestroyVDSCommand(HostName =
>athos,
>DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
>vmId='d4418a18-46e4-445b-b344-49aa10861617', secondsToWait='0',
>gracefully='false', reason='', ignoreNoVm='true'}), log id:
69aa3373
>2020-06-25 15:25:30,776-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
>(ForkJoinPool-1-worker-15) [] FINISH, DestroyVDSCommand, return: , log
>id: 69aa3373
>2020-06-25 15:25:30,776-05 INFO
>[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>(ForkJoinPool-1-worker-15) [] VM
>'d4418a18-46e4-445b-b344-49aa10861617'(aries) moved from 'Up' -->
>'Down'
>2020-06-25 15:25:30,876-05 INFO
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>(ForkJoinPool-1-worker-15) [] EVENT_ID: VM_DOWN(61), VM aries is down.
>Exit message: User shut down from within the guest
>2020-06-25 15:25:30,898-05 INFO
>[org.ovirt.engine.core.bll.ProcessDownVmCommand]
>(EE-ManagedThreadFactory-engine-Thread-22900) [4e3a6bbe] Running
>command: ProcessDownVmCommand internal: true.
>2020-06-25 15:25:31,060-05 INFO
>[org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
>(EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Fetched 2 VMs
>from VDS '279ef2ed-c140-4e57-acf6-d5da176ee237'
>2020-06-25 15:25:45,266-05 INFO
>[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
>(ForkJoinPool-1-worker-6) [] VM '632e98d0-4262-4e20-8e31-6d94e4401193'
>was reported as Down on VDS
>'279ef2ed-c140-4e57-acf6-d5da176ee237'(athos)
>2020-06-25 15:25:45,268-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
>(ForkJoinPool-1-worker-6) [] START, DestroyVDSCommand(HostName = athos,
>DestroyVmVDSCommandParameters:{hostId='279ef2ed-c140-4e57-acf6-d5da176ee237',
>vmId='632e98d0-4262-4e20-8e31-6d94e4401193', secondsToWait='0',
>gracefully='false', reason='', ignoreNoVm='true'}), log id:
2e0f4bde
>2020-06-25 15:25:45,272-05 INFO
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
>(ForkJoinPool-1-worker-6) [] FINISH, DestroyVDSCommand, return: , log
>id: 2e0f4bde
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/SDUWOYANYZ27RQDQPN22347FE4ZSEP5V/
5:54 p.m.
Here is what I found in the audit log:
on Centos 7:
type=SERVICE_START msg=audit(06/25/2020 15:24:18.967:80755) : pid=1 uid=root auid=unset
ses=unset msg='unit=systemd-shutdownd comm=systemd exe=/usr/lib/systemd/systemd
hostname=? addr=? terminal=? res=success'
type=SYSTEM_RUNLEVEL msg=audit(06/25/2020 15:25:19.501:103256) : user pid=12951 uid=root
auid=unset ses=unset msg='old-level=3 new-level=0 exe=/sbin/shutdown hostname=? addr=?
terminal=? res=success'
----
type=SYSTEM_SHUTDOWN msg=audit(06/25/2020 15:25:19.501:103257) : user pid=12951 uid=root
auid=unset ses=unset msg='init exe=/sbin/shutdown hostname=? addr=? terminal=?
res=success'
=============================================================
on Centos 6:
[root@aries ~]# ausearch -i -p 12951
----
type=USER_AUTH msg=audit(05/23/2020 20:23:44.890:8878) : user pid=12951 uid=ovirtagent
auid=unset ses=unset msg='op=PAM:authentication acct=ovirtagent
exe=/usr/sbin/userhelper hostname=? addr=? terminal=? res=success'
----
type=USER_ACCT msg=audit(05/23/2020 20:23:44.891:8879) : user pid=12951 uid=ovirtagent
auid=unset ses=unset msg='op=PAM:accounting acct=ovirtagent exe=/usr/sbin/userhelper
hostname=? addr=? terminal=? res=success'
----
type=SYSTEM_RUNLEVEL msg=audit(06/25/2020 15:25:19.501:103256) : user pid=12951 uid=root
auid=unset ses=unset msg='old-level=3 new-level=0 exe=/sbin/shutdown hostname=? addr=?
terminal=? res=success'
----
type=SYSTEM_SHUTDOWN msg=audit(06/25/2020 15:25:19.501:103257) : user pid=12951 uid=root
auid=unset ses=unset msg='init exe=/sbin/shutdown hostname=? addr=? terminal=?
res=success'
----
1605
days inactive
1607
days old
4 comments
4 participants
participants (4)
-
Antoine Nguyen -
Bobby
-
Lev Veyde -
Strahil Nikolov