[Users] All VMs disappeared

Mon Aug 12 15:59:04 UTC 2013

Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
>> I looked around Noam's patch and that should not cause such behavior. 
>> I am wondering how that lost VM's could happen.
>>
>> Jakub, can you give a more detailed description what you were doing 
>> with oVirt when this happened? Maybe the bug is still there.
>>
>> Thank you,
>> Laszlo
>>
>> ----- Original Message -----
>>> From: "Greg Sheremeta" <gshereme at redhat.com>
>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>> Cc: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko" 
>>> <nslomian at redhat.com>, users at ovirt.org
>>> Sent: Monday, August 12, 2013 1:23:40 PM
>>> Subject: Re: [Users] All VMs disappeared
>>>
>>> Not the one I fixed, 987907. It was a simple UI NullPointerException.
>>>
>>> Greg
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>> <nslomian at redhat.com>, "Greg Sheremeta"
>>>> <gshereme at redhat.com>
>>>> Cc: users at ovirt.org
>>>> Sent: Monday, August 12, 2013 7:21:26 AM
>>>> Subject: Re: [Users] All VMs disappeared
>>>>
>>>> Well if they are no longer in DB then that explains why the 
>>>> exception no
>>>> longer occurs, but at the cost of database corruption.
>>>> Noam and Greg, can these bugs cause data corruption?
>>>>
>>>> Thank you,
>>>> Laszlo
>>>>
>>>> ----- Original Message -----
>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>> To: "Greg Sheremeta" <gshereme at redhat.com>
>>>>> Cc: users at ovirt.org, "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>
>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
>>>>>> It could also be this bug[1], for which I just submitted a fix.
>>>>>>
>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
>>>>>>
>>>>>> You can work around it by typing just "VMs:" (without the quotes) in
>>>>>> the
>>>>>> search bar.
>>>>>>
>>>>>> Greg
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>> Cc: users at ovirt.org
>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>
>>>>>>> Hi Jakub,
>>>>>>>
>>>>>>> Could you check through DB or REST-API if the VM's are in your DB?
>>>>>>> select * from vm_static;
>>>>>>> or
>>>>>>> curl -u admin at internal:blablabla [engine-url]api/vms
>>>>>>>
>>>>>>> It seems Noam fixed this issue already in
>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Laszlo
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>> To: users at ovirt.org
>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>
>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs
>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. Repeating
>>>>>>>>> problem in log is:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013-08-09 09:44:24,203 WARN
>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds =
>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error =
>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>
>>>>>>>>>           at 
>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>           at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
>>>>>>>>>
>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) 
>>>>>>>>>
>>>>>>>>> [scheduler.jar:]
>>>>>>>>>           at 
>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>> [quartz.jar:]
>>>>>>>>>           at
>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 
>>>>>>>>>
>>>>>>>>> [quartz.jar:]
>>>>>>>>>
>>>>>>>>> 2013-08-09 09:44:27,039 WARN
>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds =
>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error =
>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) 
>>>>>>>>>
>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>
>>>>>>>>>           at 
>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>           at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
>>>>>>>>>
>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>           at
>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) 
>>>>>>>>>
>>>>>>>>> [scheduler.jar:]
>>>>>>>>>           at 
>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>> [quartz.jar:]
>>>>>>>>>           at
>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 
>>>>>>>>>
>>>>>>>>> [quartz.jar:]
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>> I restarted management node and both nodes and this problem
>>>>>>>> disappeared.
>>>>>>>> But I still can not see any VMs.
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>>>>> Hi,
>>>>>
>>>>> I do not have them in database. Both commands shows nothing
>>>>>
> Hello,
>
> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great 
> (except SPICE. VNC and creating new WM due to serialization bug.). 
> Because those bugs were irritating, we switched to nightly. Still 
> works fine (but bug with VMs is not fixed there), than I installed new 
> ovirt engine management server in KVM virtualization (using libvirtd) 
> due to taking snapshots before deploying nightly builds. New 
> virtualized engine works great and I delete old one. Next day I came 
> to work and engine web page was very very slow, every operation takes 
> ages, that time I found there were no VMs, no HDDs and so. I converted 
> its qcow2 image to raw, because I think it has slow DB because of 
> running i qcow2. Did not helped, than I take a look in nodes logs and 
> found this problem, which I posted in first email (VMs still running, 
> but not visible in engine VMs tab, but on host I can see number of 
> running VMs). After restart bug disappeared, but VMs too.
> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are 
> gone. Even from storage. (but disappearing from storage may be my 
> fault during recovery).
>
> I can provide logs from both two nodes and (hopefully) from engine 
> too. I can provide you with those logs, but not publicly. Sorry.
>
> Thank you,
> Jakub Bittner, NSA Czech
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again.