4:49 a.m.
Hello,
in my installation I have to use poor storage... the oVirt installation doesn't manage
such a case and begin to "balance" and move VMs around... taking too many
snapshots stressing a poor performance all the cluster mess up....
Why the vms don't go in "Pause" state but the cluster prefer to migrate
things around messing up everything?
This is a reference I found and for notice I'm disabling the auto-migration on every
VM, hoping this help
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/24KQZFP2PCW4...
5:15 a.m.
On Tue, Mar 28, 2023 at 11:50 AM Diego Ercolani <diego.ercolani(a)ssis.sm>
wrote:
Hello,
in my installation I have to use poor storage... the oVirt installation
doesn't manage such a case and begin to "balance" and move VMs around...
taking too many snapshots stressing a poor performance all the cluster mess
up....
Why the vms don't go in "Pause" state but the cluster prefer to migrate
things around messing up everything?
This is a reference I found and for notice I'm disabling the
auto-migration on every VM, hoping this help
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/24KQZFP2PCW4...
What is your current scheduling policy for the related oVirt cluster?
What is the event/error you see in engine.log of the engine and vdsm.log of
the host previously carrying on the VM when it happens?
Gianluca
5:31 a.m.
The scheduling policy was the "Suspend Workload if needed" and disabled parallel
migration.
The problem is that The Engine (mapped on external NFS domain implemented by a linux box
without any other vm mapped) simply disappear. I have a single 10Gbps intel ethernet link
that I use to distribute storage, management and "production" networks, but I
don't record any bandwidth limit issue
5:33 a.m.
I record entry like this in the journal of everynode:
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58 1191247 [4105511]: s9
delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/dom_md/ids
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58 1191247 [4105511]: s9
renewal error -202 delta_length 10 last_success 1191216
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58 1191247 [2750073]:
s11 delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/ovirt-nfsha.ovirt:_dati_drbd0/2527ed0f-e91a-4748-995c-e644362e8408/dom_md/ids
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58 1191247 [2750073]:
s11 renewal error -202 delta_length 10 last_success 1191217
as You see its complaining about a gluster volume (hosting vms and mapped on three node
with the terrible SATA SSD: Samsung_SSD_870_EVO_4TB
7:49 a.m.
On Tue, Mar 28, 2023 at 12:34 PM Diego Ercolani <diego.ercolani(a)ssis.sm>
wrote:
I record entry like this in the journal of everynode:
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58
1191247 [4105511]: s9 delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/dom_md/ids
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58
1191247 [4105511]: s9 renewal error -202 delta_length 10 last_success
1191216
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58
1191247 [2750073]: s11 delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/ovirt-nfsha.ovirt:_dati_drbd0/2527ed0f-e91a-4748-995c-e644362e8408/dom_md/ids
Mar 28 10:26:58 ovirt-node3.ovirt sanlock[1660]: 2023-03-28 10:26:58
1191247 [2750073]: s11 renewal error -202 delta_length 10 last_success
1191217
as You see its complaining about a gluster volume (hosting vms and mapped
on three node with the terrible SATA SSD: Samsung_SSD_870_EVO_4TB
And inside the engine.log file of the engine, when it becomes reachable
again?
8:41 a.m.
On Tue, Mar 28, 2023 at 3:30 PM Diego Ercolani <diego.ercolani(a)ssis.sm>
wrote:
No, now seem "stable" awaiting for next event
I mean logs around the time of arising problems... It engine has not
shutdown it will contain logs generated on it during the problematic
timeframe...
9 a.m.
It's difficult to answer as the engine normally "freezes" or is taken down
during events... I will try to get them next time
9:40 a.m.
New event:
Mar 28 14:37:32 ovirt-node3.ovirt vdsm[4288]: WARN executor state: count=5
workers={<Worker name=periodic/4 waiting task#=4 at 0x7fce0e9d6048>, <Worker
name=periodic/5 waiting task#=0 at 0x7fcda8700860>, <Worker name=periodic/3 waiting
task#=188 at 0x7fcdc0010470>, <Worker name=periodic/1 waiting task#=189 at
0x7fcdc00102b0>, <Worker name=periodic/2 running <Task discardable <Operation
action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fcdc007a048> at
0x7fcdc0010898> timeout=7.5, duration=7.50 at 0x7fcdc0010208> discarded task#=189 at
0x7fcdc0010390>}
Mar 28 14:37:32 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:32 829 [7438]: s4
delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/dom_md/ids
Mar 28 14:37:32 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:32 829 [7438]: s4
renewal error -202 delta_length 10 last_success 798
Mar 28 14:37:33 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:33 830 [7660]: s6
delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/ovirt-nfsha.ovirt:_dati_drbd0/2527ed0f-e91a-4748-995c-e644362e8408/dom_md/ids
Mar 28 14:37:33 ovirt-node3.ovirt sanlock[1662]: 2023-03-28 14:37:33 830 [7660]: s6
renewal error -202 delta_length 10 last_success 799
Mar 28 14:37:36 ovirt-node3.ovirt pacemaker-controld[3145]: notice: High CPU load
detected: 32.590000
Mar 28 14:37:36 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: We did not
send a P_BARRIER for 14436ms > ko-count (7) * timeout (10 * 0.1s); drbd kernel thread
blocked?
Mar 28 14:37:41 ovirt-node3.ovirt libvirtd[2735]: Domain id=1 name='SSIS-microos'
uuid=e41f8148-79ab-4a88-879f-894d5750e5fb is tainted: custom-ga-command
Mar 28 14:37:49 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: We did not
send a P_BARRIER for 7313ms > ko-count (7) * timeout (10 * 0.1s); drbd kernel thread
blocked?
Mar 28 14:37:56 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: We did not
send a P_BARRIER for 14481ms > ko-count (7) * timeout (10 * 0.1s); drbd kernel thread
blocked?
Mar 28 14:38:06 ovirt-node3.ovirt pacemaker-controld[3145]: notice: High CPU load
detected: 33.500000
Mar 28 14:38:09 ovirt-node3.ovirt kernel: drbd drbd0/0 drbd1 ovirt-node2.ovirt: Remote
failed to finish a request within 7010ms > ko-count (7) * timeout (10 * 0.1s)
2023-03-28 14:37:32,601Z INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-73) [] There is no
host with more than 10 running guests, no balancing is needed
2023-03-28 14:37:50,662Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] VM
'ccb06298-33a3-4b6f-bff3-d0bcd494b18d'(TpayX2GO) moved from 'Up' -->
'NotResponding'
2023-03-28 14:37:50,666Z WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-41) [] EVENT_ID:
VM_NOT_RESPONDING(126), VM TpayX2GO is not responding.
2023-03-28 14:38:01,087Z WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6602) [] domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' in problem 'PROBLEMATIC'. vds:
'ovirt-node2.ovirt'
2023-03-28 14:38:05,676Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-20) [] VM
'ccb06298-33a3-4b6f-bff3-d0bcd494b18d'(TpayX2GO) moved from
'NotResponding' --> 'Up'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6609) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' recovered from problem. vds:
'ovirt-node2.ovirt'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6609) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' recovered from problem. vds:
'ovirt-node2.ovirt'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6610) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' recovered from problem. vds:
'ovirt-node4.ovirt'
2023-03-28 14:38:16,107Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6610) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' recovered from problem. vds:
'ovirt-node4.ovirt'
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' recovered from problem. vds:
'ovirt-node3.ovirt'
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'2527ed0f-e91a-4748-995c-e644362e8408:drbd0' has recovered from problem. No active
host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' recovered from problem. vds:
'ovirt-node3.ovirt'
2023-03-28 14:38:16,327Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl]
(EE-ManagedThreadFactory-engine-Thread-6612) [] Domain
'4745320f-bfc3-46c4-8849-b4fe8f1b2de6:gv0' has recovered from problem. No active
host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2023-03-28 14:38:18,602Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [] VM
'e41f8148-79ab-4a88-879f-894d5750e5fb'(SSIS-microos) moved from 'Up'
--> 'NotResponding'
2023-03-28 14:38:18,607Z WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [] EVENT_ID:
VM_NOT_RESPONDING(126), VM SSIS-microos is not responding.
2023-03-28 14:38:32,603Z INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-32) [] There is no
host with more than 10 running guests, no balancing is needed
403
days inactive
403
days old
8 comments
2 participants
participants (2)
-
Diego Ercolani -
Gianluca Cecchi