Re: [ovirt-users] VMs are not running/booting on one host

On 15/02/2017 21:20, Nir Soffer wrote: > On Wed, Feb 15, 2017 at 10:05 PM, Peter Hudec <phudec(a)cnc.sk&gt; wrote: >> Hi, >> >> so theproblem is little bit different. When I wait for a long time, the >> VM boots ;( > > Is this an issue only with old vms imported from the old setup, or > also with new vms? I do not have new VMs, so with the OLD one.But I did not import them from old setup. The Host OS upgrade I did by our docs, creating new cluster, host upgrade and vm migrations. There was no outage until now. I tried to install new VM, but the installer hangs on that host. > >> >> But ... /see the log/. I'm invetigating the reason. >> The difference between the dipovirt0{1,2} and the dipovirt03 isthe >> installation time. The first 2 was migrated last week, the last one >> yesterday. There some newer packages, but nothing related to KVM. >> >> [ 292.429622] INFO: rcu_sched self-detected stall on CPU { 0} (t=72280 >> jiffies g=393 c=392 q=35) >> [ 292.430294] sending NMI to all CPUs: >> [ 292.430305] NMI backtrace for cpu 0 >> [ 292.430309] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-4-amd64 >> #1 Debian 3.16.39-1 >> [ 292.430311] Hardware name: oVirt oVirt Node, BIOS 0.5.1 01/01/2011 >> [ 292.430313] task: ffffffff8181a460 ti: ffffffff81800000 task.ti: >> ffffffff81800000 >> [ 292.430315] RIP: 0010:[<ffffffff81052ae6>] [<ffffffff81052ae6>] >> native_write_msr_safe+0x6/0x10 >> [ 292.430323] RSP: 0018:ffff88001fc03e08 EFLAGS: 00000046 >> [ 292.430325] RAX: 0000000000000400 RBX: 0000000000000000 RCX: >> 0000000000000830 >> [ 292.430326] RDX: 0000000000000000 RSI: 0000000000000400 RDI: >> 0000000000000830 >> [ 292.430327] RBP: ffffffff818e2a80 R08: ffffffff818e2a80 R09: >> 00000000000001e8 >> [ 292.430329] R10: 0000000000000000 R11: ffff88001fc03b96 R12: >> 0000000000000000 >> [ 292.430330] R13: 000000000000a0ea R14: 0000000000000002 R15: >> 0000000000080000 >> [ 292.430335] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000) >> knlGS:0000000000000000 >> [ 292.430337] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >> [ 292.430339] CR2: 0000000001801000 CR3: 000000001c6de000 CR4: >> 00000000000006f0 >> [ 292.430343] Stack: >> [ 292.430344] ffffffff8104b30d 0000000000000002 0000000000000082 >> ffff88001fc0d6a0 >> [ 292.430347] ffffffff81853800 0000000000000000 ffffffff818e2fe0 >> 0000000000000023 >> [ 292.430349] ffffffff81853800 ffffffff81047d63 ffff88001fc0d6a0 >> ffffffff810c73fa >> [ 292.430352] Call Trace: >> [ 292.430354] <IRQ> >> >> [ 292.430360] [<ffffffff8104b30d>] ? __x2apic_send_IPI_mask+0xad/0xe0 >> [ 292.430365] [<ffffffff81047d63>] ? >> arch_trigger_all_cpu_backtrace+0xc3/0x140 >> [ 292.430369] [<ffffffff810c73fa>] ? rcu_check_callbacks+0x42a/0x670 >> [ 292.430373] [<ffffffff8109bb1e>] ? account_process_tick+0xde/0x180 >> [ 292.430376] [<ffffffff810d1e00>] ? tick_sched_handle.isra.16+0x60/0x60 >> [ 292.430381] [<ffffffff81075fc0>] ? update_process_times+0x40/0x70 >> [ 292.430404] [<ffffffff810d1dc0>] ? tick_sched_handle.isra.16+0x20/0x60 >> [ 292.430407] [<ffffffff810d1e3c>] ? tick_sched_timer+0x3c/0x60 >> [ 292.430410] [<ffffffff8108c6a7>] ? __run_hrtimer+0x67/0x210 >> [ 292.430412] [<ffffffff8108caa9>] ? hrtimer_interrupt+0xe9/0x220 >> [ 292.430416] [<ffffffff8151dcab>] ? smp_apic_timer_interrupt+0x3b/0x50 >> [ 292.430420] [<ffffffff8151bd3d>] ? apic_timer_interrupt+0x6d/0x80 >> [ 292.430422] <EOI> >> >> [ 292.430425] [<ffffffff8109b2e5>] ? sched_clock_local+0x15/0x80 >> [ 292.430428] [<ffffffff8101da50>] ? mwait_idle+0xa0/0xa0 >> [ 292.430431] [<ffffffff81052c22>] ? native_safe_halt+0x2/0x10 >> [ 292.430434] [<ffffffff8101da69>] ? default_idle+0x19/0xd0 >> [ 292.430437] [<ffffffff810a9b74>] ? cpu_startup_entry+0x374/0x470 >> [ 292.430440] [<ffffffff81903076>] ? start_kernel+0x497/0x4a2 >> [ 292.430442] [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e >> [ 292.430445] [<ffffffff81902120>] ? early_idt_handler_array+0x120/0x120 >> [ 292.430447] [<ffffffff8190271f>] ? x86_64_start_kernel+0x14d/0x15c >> [ 292.430448] Code: c2 48 89 d0 c3 89 f9 0f 32 31 c9 48 c1 e2 20 89 c0 >> 89 0e 48 09 c2 48 89 d0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 89 f0 89 f9 >> 0f 30 <31> c0 c3 0f 1f 80 00 00 00 00 89 f9 0f 33 48 c1 e2 20 89 c0 48 >> [ 292.430579] Clocksource tsc unstable (delta = -289118137838 ns) >> >> >> On 15/02/2017 20:39, Peter Hudec wrote: >>> Hi, >>> >>> I did already, but not find any suspicious, see attached logs and the >>> spice screenshot. >>> >>> Actually the VM is booting, but is stacked in some bad state. >>> When migrating, the migration is sucessfull, but the vm is not acessible >>> /even on network/ >>> >>> Right now I found one VM, which is working well. >>> >>> In logs look for diplci01 at 2017-02-15 20:23:00,420, the VM ID is >>> 7ddf349b-fb9a-44f4-9e88-73e84625a44e >>> >>> thanks >>> Peter >>> >>> On 15/02/2017 19:40, Nir Soffer wrote: >>>> On Wed, Feb 15, 2017 at 8:11 PM, Peter Hudec <phudec(a)cnc.sk&gt; wrote: >>>>> Hi, >>>>> >>>>> I'm preparing to migrate from 3.5 to 3.6 >>>>> The first step is the CentOS6 -> CentOS7 for hosts. >>>>> >>>>> setup: >>>>> - 3x hosts /dipovitrt01, dipovirt02, dipovirt03/ >>>>> - 1x hosted engine /on all 3 hosts/ >>>>> >>>>> The upgrade of the first 2 hosts was OK, all VM are running OK. >>>>> When I upgraded the 3rd host /dipovirt03/, some VMs are not able to run >>>>> on the or boot on this host. I tried to full reinstall the host, but >>>>> wth the same result. >>>>> >>>>> In case of migration the VMm will stop running in a while. >>>>> In case of booting the VM will not boot, I see the 'Loading kernel ...' >>>>> >>>>> Almost all VMS are Debian 8 with guest tools, some Centos 6/7 >>>>> >>>>> The hosts were OK with CentOS6. >>>>> >>>>> >>>>> Where should I start to investigate ? >>>> >>>> Sharing vdsm logs showing the failed attempts to run or migrate >>>> a vm would be a good start. >>>> >>>>> >>>>> best regards >>>>> Peter >>>>> >>>>> >>>>> -- >>>>> *Peter Hudec* >>>>> Infraštruktúrny architekt >>>>> phudec(a)cnc.sk <mailto:phudec@cnc.sk> >>>>> >>>>> *CNC, a.s.* >>>>> Borská 6, 841 04 Bratislava >>>>> Recepcia: +421 2 35 000 100 >>>>> >>>>> Mobil:+421 905 997 203 >>>>> *www.cnc.sk* <http:///www.cnc.sk> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> >> -- >> *Peter Hudec* >> Infraštruktúrny architekt >> phudec(a)cnc.sk <mailto:phudec@cnc.sk> >> >> *CNC, a.s.* >> Borská 6, 841 04 Bratislava >> Recepcia: +421 2 35 000 100 >> >> Mobil:+421 905 997 203 >> *www.cnc.sk* <http:///www.cnc.sk> >>