|
||||||||||||||
Dear List,
we upgraded our OVirt 4.4.9 Infrastructure (engine and hosts) to
latest available 4.4.10.x. Engine shows "4.4.10.7-1.el8" on Login
Page
Hosts are based on ovirt-node-ng (nodectl info outputs ovirt-node-ng-4.4.10.2-0.20220303.0)
When we migrate VM's we see VM's dying short after migration with below errors also printed to console. We are only able to shutdown these VM's (PowerOff) and start them up again. Mostly they are started again on the Host that was target of migration and they run without issues then.
All Servers are EPYC based, but different CPU Versions. All hosts are in a single "Cluster", CPU Type is "Secure AMD EPYC". With 4.4.8 and 4.4.9 we had no problems migrating VM's between hosts.
Cluster is:
2 Hosts: AMD EPYC 7401P 24-Core Processor
1 Hosts: AMD EPYC 7402P 24-Core Processor
7 Hosts: AMD EPYC 7502P 32-Core Processor
What can we do ? As said, we didn't see such problems for 1 year
running 4.4.8.x and 4.4.9.x.
Syslog showing crash and date/clock error or sudden offset:
Jan 19 19:23:22 myvmXX kernel: [163933.218848] rcu: INFO:
rcu_sched self-detected stall on CPU
Jan 19 19:23:22 myvmXX kernel: [163933.218876] rcu: 1-...!:
(8 GPs behind) idle=78a/0/0x1 softirq=4456350/4456351 fqs=0
Jan 19 19:23:22 myvmXX kernel: [163933.218901] (t=537752 jiffies
g=9505317 q=22)
Jan 19 19:23:22 myvmXX kernel: [163933.218903] rcu: rcu_sched
kthread starved for 537752 jiffies! g9505317 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
Jan 19 19:23:22 myvmXX kernel: [163933.218932] rcu: RCU
grace-period kthread stack dump:
Jan 19 19:23:22 myvmXX kernel: [163933.218949] rcu_sched
I 0 10 2 0x80004000
Jan 19 19:23:22 myvmXX kernel: [163933.218951] Call Trace:
Jan 19 19:23:22 myvmXX kernel: [163933.218957]
__schedule+0x2e3/0x740
Jan 19 19:23:22 myvmXX kernel: [163933.218960] schedule+0x42/0xb0
Jan 19 19:23:22 myvmXX kernel: [163933.218961]
schedule_timeout+0x8a/0x160
Jan 19 19:23:22 myvmXX kernel: [163933.218964] ?
rcu_accelerate_cbs+0x28/0x190
Jan 19 19:23:22 myvmXX kernel: [163933.218967] ?
__next_timer_interrupt+0xe0/0xe0
Jan 19 19:23:22 myvmXX kernel: [163933.218969]
rcu_gp_kthread+0x48d/0x9a0
Jan 19 19:23:22 myvmXX kernel: [163933.218971]
kthread+0x104/0x140
Jan 19 19:23:22 myvmXX kernel: [163933.218972] ?
kfree_call_rcu+0x20/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.218974] ?
kthread_park+0x90/0x90
Jan 19 19:23:22 myvmXX kernel: [163933.218975]
ret_from_fork+0x35/0x40
Jan 19 19:23:22 myvmXX kernel: [163933.218982] Sending NMI from
CPU 1 to CPUs 0:
Jan 19 19:23:22 myvmXX kernel: [163933.219978] NMI backtrace for
cpu 0
Jan 19 19:23:22 myvmXX kernel: [163933.219978] CPU: 0 PID: 0 Comm:
swapper/0 Not tainted 5.4.0-137-generic #154-Ubuntu
Jan 19 19:23:22 myvmXX kernel: [163933.219979] Hardware name:
oVirt RHEL, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
Jan 19 19:23:22 myvmXX kernel: [163933.219979] RIP:
0010:timekeeping_advance+0x12f/0x5a0
Jan 19 19:23:22 myvmXX kernel: [163933.219980] Code: 00 48 8b 35
e3 a0 ec 01 bb 00 ca 9a 3b 49 29 c7 48 01 05 04 a0 ec 01 48 01 05
35 a0 ec 01 48 89 f2 48 d3 e2 8b 0d fd 9f ec 01 <48> 03 15
fa 9f ec 01 48 89 15 f3 9f ec 01 48 d3 e3 48 39 da 72 57
Jan 19 19:23:22 myvmXX kernel: [163933.219981] RSP:
0018:ffffb5a580003e40 EFLAGS: 00000016
Jan 19 19:23:22 myvmXX kernel: [163933.219981] RAX:
000000007a120000 RBX: 000000003b9aca00 RCX: 0000000000000017
Jan 19 19:23:22 myvmXX kernel: [163933.219982] RDX:
003d091a39de0000 RSI: 00001e848d1cef00 RDI: 00000000003d0900
Jan 19 19:23:22 myvmXX kernel: [163933.219982] RBP:
ffffb5a580003e98 R08: 0000000000000000 R09: 666657f4d876bbc2
Jan 19 19:23:22 myvmXX kernel: [163933.219983] R10:
ffff8d1ee8119618 R11: ffff8d1effa2ffb8 R12: 0000000000000000
Jan 19 19:23:22 myvmXX kernel: [163933.219983] R13:
0000000000000000 R14: 0000000000000009 R15: 6666232f185ebbc2
Jan 19 19:23:22 myvmXX kernel: [163933.219984] FS:
0000000000000000(0000) GS:ffff8d1effa00000(0000)
knlGS:0000000000000000
Jan 19 19:23:22 myvmXX kernel: [163933.219984] CS: 0010 DS: 0000
ES: 0000 CR0: 0000000080050033
Jan 19 19:23:22 myvmXX kernel: [163933.219984] CR2:
000055a093634da0 CR3: 00000001280d6000 CR4: 00000000003406f0
Jan 19 19:23:22 myvmXX kernel: [163933.219985] Call Trace:
Jan 19 19:23:22 myvmXX kernel: [163933.219985] <IRQ>
Jan 19 19:23:22 myvmXX kernel: [163933.219985] ?
ttwu_do_activate+0x5b/0x70
Jan 19 19:23:22 myvmXX kernel: [163933.219985]
update_wall_time+0x10/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.219986]
tick_do_update_jiffies64.part.0+0x88/0xd0
Jan 19 19:23:22 myvmXX kernel: [163933.219986]
tick_sched_do_timer+0x58/0x60
Jan 19 19:23:22 myvmXX kernel: [163933.219986]
tick_sched_timer+0x2d/0x80
Jan 19 19:23:22 myvmXX kernel: [163933.219987]
__hrtimer_run_queues+0xf7/0x270
Jan 19 19:23:22 myvmXX kernel: [163933.219987] ?
tick_sched_do_timer+0x60/0x60
Jan 19 19:23:22 myvmXX kernel: [163933.219987]
hrtimer_interrupt+0x109/0x220
Jan 19 19:23:22 myvmXX kernel: [163933.219988]
smp_apic_timer_interrupt+0x71/0x140
Jan 19 19:23:22 myvmXX kernel: [163933.219988]
apic_timer_interrupt+0xf/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.219988] </IRQ>
Jan 19 19:23:22 myvmXX kernel: [163933.219988] RIP:
0010:native_safe_halt+0xe/0x10
Jan 19 19:23:22 myvmXX kernel: [163933.219989] Code: 7b ff ff ff
eb bd 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d d6 39 51 00 f4 c3
66 90 e9 07 00 00 00 0f 00 2d c6 39 51 00 fb f4 <c3> 90 0f
1f 44 00 00 55 48 89 e5 41 55 41 54 53 e8 9d 5e 62 ff 65
Jan 19 19:23:22 myvmXX kernel: [163933.219990] RSP:
0018:ffffffff8be03e18 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Jan 19 19:23:22 myvmXX kernel: [163933.219990] RAX:
ffffffff8aef7a20 RBX: 0000000000000000 RCX: 0000000000000001
Jan 19 19:23:22 myvmXX kernel: [163933.219991] RDX:
000000000bc48666 RSI: ffffffff8be03dd8 RDI: 00009519477dc7a5
Jan 19 19:23:22 myvmXX kernel: [163933.219991] RBP:
ffffffff8be03e38 R08: 0000000000000001 R09: 0000000000000002
Jan 19 19:23:22 myvmXX kernel: [163933.219992] R10:
0000000000000000 R11: 0000000000000001 R12: 0000000000000000
Jan 19 19:23:22 myvmXX kernel: [163933.219992] R13:
ffffffff8be13780 R14: 0000000000000000 R15: 0000000000000000
Jan 19 19:23:22 myvmXX kernel: [163933.219992] ?
__cpuidle_text_start+0x8/0x8
Jan 19 19:23:22 myvmXX kernel: [163933.219993] ?
tick_nohz_idle_stop_tick+0x164/0x290
Jan 19 19:23:22 myvmXX kernel: [163933.219993] ?
default_idle+0x20/0x140
Jan 19 19:23:22 myvmXX kernel: [163933.219993]
arch_cpu_idle+0x15/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.219994]
default_idle_call+0x23/0x30
Jan 19 19:23:22 myvmXX kernel: [163933.219994]
do_idle+0x1fb/0x270
Jan 19 19:23:22 myvmXX kernel: [163933.219994]
cpu_startup_entry+0x20/0x30
Jan 19 19:23:22 myvmXX kernel: [163933.219994]
rest_init+0xae/0xb0
Jan 19 19:23:22 myvmXX kernel: [163933.219995]
arch_call_rest_init+0xe/0x1b
Jan 19 19:23:22 myvmXX kernel: [163933.219995]
start_kernel+0x52f/0x550
Jan 19 19:23:22 myvmXX kernel: [163933.219995]
x86_64_start_reservations+0x24/0x26
Jan 19 19:23:22 myvmXX kernel: [163933.219996]
x86_64_start_kernel+0x8f/0x93
Jan 19 19:23:22 myvmXX kernel: [163933.219996]
secondary_startup_64+0xa4/0xb0
Jan 19 19:23:22 myvmXX kernel: [163933.220000] NMI backtrace for
cpu 1
Jan 19 19:23:22 myvmXX kernel: [163933.220002] CPU: 1 PID: 0 Comm:
swapper/1 Not tainted 5.4.0-137-generic #154-Ubuntu
Jan 19 19:23:22 myvmXX kernel: [163933.220003] Hardware name:
oVirt RHEL, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
Jan 19 19:23:22 myvmXX kernel: [163933.220003] Call Trace:
Jan 19 19:23:22 myvmXX kernel: [163933.220004] <IRQ>
Jan 19 19:23:22 myvmXX kernel: [163933.220006]
dump_stack+0x6d/0x8b
Jan 19 19:23:22 myvmXX kernel: [163933.220008] ?
lapic_can_unplug_cpu+0x80/0x80
Jan 19 19:23:22 myvmXX kernel: [163933.220010]
nmi_cpu_backtrace.cold+0x14/0x53
Jan 19 19:23:22 myvmXX kernel: [163933.220013]
nmi_trigger_cpumask_backtrace+0xe8/0xf0
Jan 19 19:23:22 myvmXX kernel: [163933.220015]
arch_trigger_cpumask_backtrace+0x19/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.220016]
rcu_dump_cpu_stacks+0x99/0xcb
Jan 19 19:23:22 myvmXX kernel: [163933.220018]
rcu_sched_clock_irq.cold+0x1b0/0x39c
Jan 19 19:23:22 myvmXX kernel: [163933.220020]
update_process_times+0x2c/0x60
Jan 19 19:23:22 myvmXX kernel: [163933.220021]
tick_sched_handle+0x29/0x60
Jan 19 19:23:22 myvmXX kernel: [163933.220022]
tick_sched_timer+0x3d/0x80
Jan 19 19:23:22 myvmXX kernel: [163933.220024]
__hrtimer_run_queues+0xf7/0x270
Jan 19 19:23:22 myvmXX kernel: [163933.220026] ?
tick_sched_do_timer+0x60/0x60
Jan 19 19:23:22 myvmXX kernel: [163933.220027]
hrtimer_interrupt+0x109/0x220
Jan 19 19:23:22 myvmXX kernel: [163933.220029]
smp_apic_timer_interrupt+0x71/0x140
Jan 19 19:23:22 myvmXX kernel: [163933.220030]
apic_timer_interrupt+0xf/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.220031] </IRQ>
Jan 19 19:23:22 myvmXX kernel: [163933.220033] RIP:
0010:native_safe_halt+0xe/0x10
Jan 19 19:23:22 myvmXX kernel: [163933.220035] Code: 7b ff ff ff
eb bd 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d d6 39 51 00 f4 c3
66 90 e9 07 00 00 00 0f 00 2d c6 39 51 00 fb f4 <c3> 90 0f
1f 44 00 00 55 48 89 e5 41 55 41 54 53 e8 9d 5e 62 ff 65
Jan 19 19:23:22 myvmXX kernel: [163933.220036] RSP:
0018:ffffb5a58007be70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Jan 19 19:23:22 myvmXX kernel: [163933.220037] RAX:
ffffffff8aef7a20 RBX: 0000000000000001 RCX: 0000000000000001
Jan 19 19:23:22 myvmXX kernel: [163933.220038] RDX:
000000000d1b7786 RSI: ffffb5a58007be30 RDI: 000095194740bea5
Jan 19 19:23:22 myvmXX kernel: [163933.220039] RBP:
ffffb5a58007be90 R08: 0000000000000001 R09: 0000000000000000
Jan 19 19:23:22 myvmXX kernel: [163933.220040] R10:
ffff8d1effa5c848 R11: 0000000000000000 R12: 0000000000000001
Jan 19 19:23:22 myvmXX kernel: [163933.220040] R13:
ffff8d1eff25af00 R14: 0000000000000000 R15: 0000000000000000
Jan 19 19:23:22 myvmXX kernel: [163933.220042] ?
__cpuidle_text_start+0x8/0x8
Jan 19 19:23:22 myvmXX kernel: [163933.220044] ?
default_idle+0x20/0x140
Jan 19 19:23:22 myvmXX kernel: [163933.220046]
arch_cpu_idle+0x15/0x20
Jan 19 19:23:22 myvmXX kernel: [163933.220047]
default_idle_call+0x23/0x30
Jan 19 19:23:22 myvmXX kernel: [163933.220049]
do_idle+0x1fb/0x270
Jan 19 19:23:22 myvmXX kernel: [163933.220051]
cpu_startup_entry+0x20/0x30
Jan 19 19:23:22 myvmXX kernel: [163933.220053]
start_secondary+0x167/0x1c0
Jan 19 19:23:22 myvmXX kernel: [163933.220054]
secondary_startup_64+0xa4/0xb0
Jan 19 19:24:34 myvmXX systemd[1]: systemd-udevd.service: Watchdog
timeout (limit 3min)!
Jan 19 19:24:34 myvmXX systemd[1]: systemd-udevd.service: Killing
process 421 (systemd-udevd) with signal SIGABRT.
Nov 15 18:38:49 myvmXX kernel: [164024.594313] rcu: INFO:
rcu_sched self-detected stall on CPU
Nov 15 18:38:49 myvmXX kernel: [164024.594352] rcu: 0-...!:
(2 ticks this GP) idle=66a/0/0x1 softirq=4017386/4017386 fqs=0
Nov 15 18:38:49 myvmXX kernel: [164024.594380] (t=1844682531753
jiffies g=9505317 q=675)
Nov 15 18:38:49 myvmXX kernel: [164024.594382] rcu: rcu_sched
kthread starved for 1844682531753 jiffies! g9505317 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=0
Nov 15 18:38:49 myvmXX kernel: [164024.594413] rcu: RCU
grace-period kthread stack dump:
Nov 15 18:38:49 myvmXX kernel: [164024.594432] rcu_sched
R 0 10 2 0x80004000
Nov 15 18:38:49 myvmXX kernel: [164024.594435] Call Trace:
Nov 15 18:38:49 myvmXX kernel: [164024.594445]
__schedule+0x2e3/0x740
Nov 15 18:38:49 myvmXX kernel: [164024.594447] schedule+0x42/0xb0
Nov 15 18:38:49 myvmXX kernel: [164024.594449]
schedule_timeout+0x8a/0x160
Nov 15 18:38:49 myvmXX kernel: [164024.594453] ?
rcu_accelerate_cbs+0x28/0x190
Nov 15 18:38:49 myvmXX kernel: [164024.594456] ?
__next_timer_interrupt+0xe0/0xe0
Nov 15 18:38:49 myvmXX kernel: [164024.594458]
rcu_gp_kthread+0x48d/0x9a0
Nov 15 18:38:49 myvmXX kernel: [164024.594460]
kthread+0x104/0x140
Nov 15 18:38:49 myvmXX kernel: [164024.594462] ?
kfree_call_rcu+0x20/0x20
Nov 15 18:38:49 myvmXX kernel: [164024.594463] ?
kthread_park+0x90/0x90
Nov 15 18:38:49 myvmXX kernel: [164024.594464]
ret_from_fork+0x35/0x40
Nov 15 18:38:49 myvmXX kernel: [164024.594467] NMI backtrace for
cpu 0
Nov 15 18:38:49 myvmXX kernel: [164024.594470] CPU: 0 PID: 0 Comm:
swapper/0 Not tainted 5.4.0-137-generic #154-Ubuntu
Nov 15 18:38:49 myvmXX kernel: [164024.594471] Hardware name:
oVirt RHEL, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
Nov 15 18:38:49 myvmXX kernel: [164024.594472] Call Trace:
Nov 15 18:38:49 myvmXX kernel: [164024.594473] <IRQ>
Nov 15 18:38:49 myvmXX kernel: [164024.594476]
dump_stack+0x6d/0x8b
Nov 15 18:38:49 myvmXX kernel: [164024.594479] ?
lapic_can_unplug_cpu+0x80/0x80
Nov 15 18:38:49 myvmXX kernel: [164024.594480]
nmi_cpu_backtrace.cold+0x14/0x53
Nov 15 18:38:49 myvmXX kernel: [164024.594484]
nmi_trigger_cpumask_backtrace+0xe8/0xf0
Nov 15 18:38:49 myvmXX kernel: [164024.594485]
arch_trigger_cpumask_backtrace+0x19/0x20
Nov 15 18:38:49 myvmXX kernel: [164024.594488]
rcu_dump_cpu_stacks+0x99/0xcb
Nov 15 18:38:49 myvmXX kernel: [164024.594489]
rcu_sched_clock_irq.cold+0x1b0/0x39c
Nov 15 18:38:49 myvmXX kernel: [164024.594491]
update_process_times+0x2c/0x60
Nov 15 18:38:49 myvmXX kernel: [164024.594494]
tick_sched_handle+0x29/0x60
Nov 15 18:38:49 myvmXX kernel: [164024.594495]
tick_sched_timer+0x3d/0x80
Nov 15 18:38:49 myvmXX kernel: [164024.594497]
__hrtimer_run_queues+0xf7/0x270
Nov 15 18:38:49 myvmXX kernel: [164024.594498] ?
tick_sched_do_timer+0x60/0x60
Nov 15 18:38:49 myvmXX kernel: [164024.594500]
hrtimer_interrupt+0x109/0x220
Nov 15 18:38:49 myvmXX kernel: [164024.594503]
smp_apic_timer_interrupt+0x71/0x140
Nov 15 18:38:49 myvmXX kernel: [164024.594504]
apic_timer_interrupt+0xf/0x20
Nov 15 18:38:49 myvmXX kernel: [164024.594505] </IRQ>
Nov 15 18:38:49 myvmXX kernel: [164024.594507] RIP:
0010:native_safe_halt+0xe/0x10
Nov 15 18:38:49 myvmXX kernel: [164024.594510] Code: 7b ff ff ff
eb bd 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d d6 39 51 00 f4 c3
66 90 e9 07 00 00 00 0f 00 2d c6 39 51 00 fb f4 <c3> 90 0f
1f 44 00 00 55 48 89 e5 41 55 41 54 53 e8 9d 5e 62 ff 65
Nov 15 18:38:49 myvmXX kernel: [164024.594511] RSP:
0018:ffffffff8be03e18 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Nov 15 18:38:49 myvmXX kernel: [164024.594513] RAX:
ffffffff8aef7a20 RBX: 0000000000000000 RCX: 0000000000000001
Nov 15 18:38:49 myvmXX kernel: [164024.594513] RDX:
000000000bc48666 RSI: ffffffff8be03dd8 RDI: 00009519477dc7a5
Nov 15 18:38:49 myvmXX kernel: [164024.594514] RBP:
ffffffff8be03e38 R08: 0000000000000001 R09: 0000000000000002
Nov 15 18:38:49 myvmXX kernel: [164024.594515] R10:
0000000000000000 R11: 0000000000000001 R12: 0000000000000000
Nov 15 18:38:49 myvmXX kernel: [164024.594515] R13:
ffffffff8be13780 R14: 0000000000000000 R15: 0000000000000000
Nov 15 18:38:49 myvmXX kernel: [164024.594517] ?
__cpuidle_text_start+0x8/0x8
Nov 15 18:38:49 myvmXX kernel: [164024.594519] ?
tick_nohz_idle_stop_tick+0x164/0x290
Nov 15 18:38:49 myvmXX kernel: [164024.594521] ?
default_idle+0x20/0x140
Nov 15 18:38:49 myvmXX kernel: [164024.594524]
arch_cpu_idle+0x15/0x20
Nov 15 18:38:49 myvmXX kernel: [164024.594525]
default_idle_call+0x23/0x30
Nov 15 18:38:49 myvmXX kernel: [164024.594528]
do_idle+0x1fb/0x270
Nov 15 18:38:49 myvmXX kernel: [164024.594530]
cpu_startup_entry+0x20/0x30
Nov 15 18:38:49 myvmXX kernel: [164024.594532]
rest_init+0xae/0xb0
Nov 15 18:38:49 myvmXX kernel: [164024.594536]
arch_call_rest_init+0xe/0x1b
Nov 15 18:38:49 myvmXX kernel: [164024.594537]
start_kernel+0x52f/0x550
Nov 15 18:38:49 myvmXX kernel: [164024.594539]
x86_64_start_reservations+0x24/0x26
Nov 15 18:38:49 myvmXX kernel: [164024.594541]
x86_64_start_kernel+0x8f/0x93
Nov 15 18:38:49 myvmXX kernel: [164024.594544]
secondary_startup_64+0xa4/0xb0
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jan
19 19:28:51 myvmXX systemd-sysctl[413]: Not setting
net/ipv4/conf/all/promote_secondaries (explicit setting exists).
|
||||||||||||||