[Users] All VMs disappeared

Itamar Heim iheim at redhat.com
Tue Aug 13 13:39:55 UTC 2013 

On 08/13/2013 08:50 AM, Jakub Bittner wrote:
> Dne 12.8.2013 22:44, Itamar Heim napsal(a):
>> On 08/12/2013 06:59 PM, Jakub Bittner wrote:
>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
>>>>> I looked around Noam's patch and that should not cause such behavior.
>>>>> I am wondering how that lost VM's could happen.
>>>>>
>>>>> Jakub, can you give a more detailed description what you were doing
>>>>> with oVirt when this happened? Maybe the bug is still there.
>>>>>
>>>>> Thank you,
>>>>> Laszlo
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>> Cc: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>> <nslomian at redhat.com>, users at ovirt.org
>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM
>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>
>>>>>> Not the one I fixed, 987907. It was a simple UI NullPointerException.
>>>>>>
>>>>>> Greg
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>> <nslomian at redhat.com>, "Greg Sheremeta"
>>>>>>> <gshereme at redhat.com>
>>>>>>> Cc: users at ovirt.org
>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM
>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>
>>>>>>> Well if they are no longer in DB then that explains why the
>>>>>>> exception no
>>>>>>> longer occurs, but at the cost of database corruption.
>>>>>>> Noam and Greg, can these bugs cause data corruption?
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Laszlo
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>> To: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>>> Cc: users at ovirt.org, "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>
>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
>>>>>>>>> It could also be this bug[1], for which I just submitted a fix.
>>>>>>>>>
>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
>>>>>>>>>
>>>>>>>>> You can work around it by typing just "VMs:" (without the
>>>>>>>>> quotes) in
>>>>>>>>> the
>>>>>>>>> search bar.
>>>>>>>>>
>>>>>>>>> Greg
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>
>>>>>>>>>> Hi Jakub,
>>>>>>>>>>
>>>>>>>>>> Could you check through DB or REST-API if the VM's are in your
>>>>>>>>>> DB?
>>>>>>>>>> select * from vm_static;
>>>>>>>>>> or
>>>>>>>>>> curl -u admin at internal:blablabla [engine-url]api/vms
>>>>>>>>>>
>>>>>>>>>> It seems Noam fixed this issue already in
>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>> Laszlo
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>> To: users at ovirt.org
>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>
>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs
>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too.
>>>>>>>>>>>> Repeating
>>>>>>>>>>>> problem in log is:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN
>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS ,
>>>>>>>>>>>> vds =
>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error =
>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>
>>>>>>>>>>>>           at
>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>           at
>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>
>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN
>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS ,
>>>>>>>>>>>> vds =
>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error =
>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>
>>>>>>>>>>>>           at
>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>           at
>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>           at
>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>> I restarted management node and both nodes and this problem
>>>>>>>>>>> disappeared.
>>>>>>>>>>> But I still can not see any VMs.
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at ovirt.org
>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I do not have them in database. Both commands shows nothing
>>>>>>>>
>>>> Hello,
>>>>
>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great
>>>> (except SPICE. VNC and creating new WM due to serialization bug.).
>>>> Because those bugs were irritating, we switched to nightly. Still
>>>> works fine (but bug with VMs is not fixed there), than I installed new
>>>> ovirt engine management server in KVM virtualization (using libvirtd)
>>>> due to taking snapshots before deploying nightly builds. New
>>>> virtualized engine works great and I delete old one. Next day I came
>>>> to work and engine web page was very very slow, every operation takes
>>>> ages, that time I found there were no VMs, no HDDs and so. I converted
>>>> its qcow2 image to raw, because I think it has slow DB because of
>>>> running i qcow2. Did not helped, than I take a look in nodes logs and
>>>> found this problem, which I posted in first email (VMs still running,
>>>> but not visible in engine VMs tab, but on host I can see number of
>>>> running VMs). After restart bug disappeared, but VMs too.
>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are
>>>> gone. Even from storage. (but disappearing from storage may be my
>>>> fault during recovery).
>>>>
>>>> I can provide logs from both two nodes and (hopefully) from engine
>>>> too. I can provide you with those logs, but not publicly. Sorry.
>>>>
>>>> Thank you,
>>>> Jakub Bittner, NSA Czech
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone
>>> again.
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>> can you verify the db from before the upgrade has the VMs?
>
> I dont know what you mean by verify database. Please, could you point me
> how to do it? Thank you
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

I assume from your expectation to see the VMs from the previous install 
that you upgraded your system rather than do a clean install?
in an upgrade, there should be a backup of the db from before the upgrade.
(in a clean install, engine is not aware of VMs running on hosts from 
previous installs - we need to cover "import existing storage domain" in 
a future version to close this gap. current alternatives around this are 
to convert the storage domain to an export domain (for an nfs data 
domain) an re-import them to the system or a few other options.

how many VMs are there?

More information about the Users mailing list