5:40 p.m.
New event:
Mar 28 14:37:32 ovirt-node3.ovirt vdsm[4288]: WARN executor state: count=5
workers={<Worker name=periodic/4 waiting task#=4 at 0x7fce0e9d6048>, <Worker
name=periodic/5 waiting task#=0 at 0x7fcda8700860>, <Worker name=periodic/3 waiting
task#=188 at 0x7fcdc0010470>, <Worker name=periodic/1 waiting task#=189 at
0x7fcdc00102b0>, <Worker name=periodic/2 running <Task discardable <Operation
action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fcdc007a048> at
0x7fcdc0010898> timeout=7.5, duration=7.50 at 0x7fcdc0010208> discarded task#=189 at
0x7fcdc0010390>}
Mar 28 14:37:32 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:32 829 [7438]: s4
delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/dom_md/ids
Mar 28 14:37:32 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:32 829 [7438]: s4
renewal error -202 delta_length 10 last_success 798
Mar 28 14:37:33 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:33 830 [7660]: s6
delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/ovirt-nfsha.ovirt:_dati_drbd0/2527ed0f-e91a-4748-995c-e644362e8408/dom_md/ids
Mar 28 14:37:33 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:33 830 [7660]: s6
renewal error -202 delta_length 10 last_success 799
Mar 28 14:37:36 ovirt-node3.ovirt pacemaker-controld[3145]: notice: High CPU load
detected: 32.590000
Mar 28 14:37:36 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: We did not
send a P_BARRIER for 14436ms > ko-count (7) * timeout (10 * 0.1s); drbd kernel thread
blocked?
Mar 28 14:37:41 ovirt-node3.ovirt libvirtd[2735]: Domain id=1 name='SSIS-microos'
uuid=e41f8148-79ab-4a88-879f-894d5750e5fb is tainted: custom-ga-command
Mar 28 14:37:49 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: We did not
send a P_BARRIER for 7313ms > ko-count (7) * timeout (10 * 0.1s); drbd kernel thread
blocked?
Mar 28 14:37:56 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: We did not
send a P_BARRIER for 14481ms > ko-count (7) * timeout (10 * 0.1s); drbd kernel thread
blocked?
Mar 28 14:38:06 ovirt-node3.ovirt pacemaker-controld[3145]: notice: High CPU load
detected: 33.500000
Mar 28 14:38:09 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: Remote
failed to finish a request within 7010ms > ko-count (7) * timeout (10 * 0.1s)
2023-03-28 14:37:32,601Z INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-73) [] There is no
host with more than 10 running guests, no balancing is needed
2023-03-28 14:37:50,662Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] VM
'ccb06298-33a3-4b6f-bff3-d0bcd494b18d'(TpayX2GO) moved from 'Up' -->
'NotResponding'
2023-03-28 14:37:50,666Z WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID:
VM_NOT_RESPONDING(126), VM TpayX2GO is not responding.
2023-03-28 14:38:01,087Z WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6602) [] domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' in problem 'PROBLEMATIC'. vds:
'ovirt-node2.ovirt'
2023-03-28 14:38:05,676Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [] VM
'ccb06298-33a3-4b6f-bff3-d0bcd494b18d'(TpayX2GO) moved from
'NotResponding' --> 'Up'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6609) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' recovered from problem. vds:
'ovirt-node2.ovirt'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6609) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' recovered from problem. vds:
'ovirt-node2.ovirt'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6610) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' recovered from problem. vds:
'ovirt-node4.ovirt'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6610) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' recovered from problem. vds:
'ovirt-node4.ovirt'
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' recovered from problem. vds:
'ovirt-node3.ovirt'
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' has recovered from problem. No active
host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' recovered from problem. vds:
'ovirt-node3.ovirt'
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' has recovered from problem. No active
host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2023-03-28 14:38:18,602Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [] VM
'e41f8148-79ab-4a88-879f-894d5750e5fb'(SSIS-microos) moved from 'Up'
--> 'NotResponding'
2023-03-28 14:38:18,607Z WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [] EVENT_ID:
VM_NOT_RESPONDING(126), VM SSIS-microos is not responding.
2023-03-28 14:38:32,603Z INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] There is no
host with more than 10 running guests, no balancing is needed