[Users] All VMs disappeared

Tue Aug 13 11:20:29 EDT 2013

Dne 13.8.2013 17:02, Itamar Heim napsal(a):
> On 08/13/2013 05:31 PM, Jakub Bittner wrote:
>> Dne 13.8.2013 15:39, Itamar Heim napsal(a):
>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote:
>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a):
>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote:
>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
>>>>>>>> I looked around Noam's patch and that should not cause such
>>>>>>>> behavior.
>>>>>>>> I am wondering how that lost VM's could happen.
>>>>>>>>
>>>>>>>> Jakub, can you give a more detailed description what you were 
>>>>>>>> doing
>>>>>>>> with oVirt when this happened? Maybe the bug is still there.
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>> Laszlo
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>> Cc: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>>>> <nslomian at redhat.com>, users at ovirt.org
>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM
>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>
>>>>>>>>> Not the one I fixed, 987907. It was a simple UI
>>>>>>>>> NullPointerException.
>>>>>>>>>
>>>>>>>>> Greg
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>>>>> <nslomian at redhat.com>, "Greg Sheremeta"
>>>>>>>>>> <gshereme at redhat.com>
>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM
>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>
>>>>>>>>>> Well if they are no longer in DB then that explains why the
>>>>>>>>>> exception no
>>>>>>>>>> longer occurs, but at the cost of database corruption.
>>>>>>>>>> Noam and Greg, can these bugs cause data corruption?
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>> Laszlo
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>> To: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>>>>>> Cc: users at ovirt.org, "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>
>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
>>>>>>>>>>>> It could also be this bug[1], for which I just submitted a 
>>>>>>>>>>>> fix.
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
>>>>>>>>>>>>
>>>>>>>>>>>> You can work around it by typing just "VMs:" (without the
>>>>>>>>>>>> quotes) in
>>>>>>>>>>>> the
>>>>>>>>>>>> search bar.
>>>>>>>>>>>>
>>>>>>>>>>>> Greg
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jakub,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in 
>>>>>>>>>>>>> your
>>>>>>>>>>>>> DB?
>>>>>>>>>>>>> select * from vm_static;
>>>>>>>>>>>>> or
>>>>>>>>>>>>> curl -u admin at internal:blablabla [engine-url]api/vms
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems Noam fixed this issue already in
>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>> Laszlo
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>>>>> To: users at ovirt.org
>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs
>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too.
>>>>>>>>>>>>>>> Repeating
>>>>>>>>>>>>>>> problem in log is:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN
>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS ,
>>>>>>>>>>>>>>> vds =
>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error =
>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>           at 
>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN
>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS ,
>>>>>>>>>>>>>>> vds =
>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error =
>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>           at 
>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>           at
>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>> I restarted management node and both nodes and this problem
>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>> But I still can not see any VMs.
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I do not have them in database. Both commands shows nothing
>>>>>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great
>>>>>>> (except SPICE. VNC and creating new WM due to serialization bug.).
>>>>>>> Because those bugs were irritating, we switched to nightly. Still
>>>>>>> works fine (but bug with VMs is not fixed there), than I installed
>>>>>>> new
>>>>>>> ovirt engine management server in KVM virtualization (using 
>>>>>>> libvirtd)
>>>>>>> due to taking snapshots before deploying nightly builds. New
>>>>>>> virtualized engine works great and I delete old one. Next day I 
>>>>>>> came
>>>>>>> to work and engine web page was very very slow, every operation 
>>>>>>> takes
>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I
>>>>>>> converted
>>>>>>> its qcow2 image to raw, because I think it has slow DB because of
>>>>>>> running i qcow2. Did not helped, than I take a look in nodes 
>>>>>>> logs and
>>>>>>> found this problem, which I posted in first email (VMs still 
>>>>>>> running,
>>>>>>> but not visible in engine VMs tab, but on host I can see number of
>>>>>>> running VMs). After restart bug disappeared, but VMs too.
>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are
>>>>>>> gone. Even from storage. (but disappearing from storage may be my
>>>>>>> fault during recovery).
>>>>>>>
>>>>>>> I can provide logs from both two nodes and (hopefully) from engine
>>>>>>> too. I can provide you with those logs, but not publicly. Sorry.
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Jakub Bittner, NSA Czech
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone
>>>>>> again.
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>> can you verify the db from before the upgrade has the VMs?
>>>>
>>>> I dont know what you mean by verify database. Please, could you 
>>>> point me
>>>> how to do it? Thank you
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>> I assume from your expectation to see the VMs from the previous
>>> install that you upgraded your system rather than do a clean install?
>>> in an upgrade, there should be a backup of the db from before the
>>> upgrade.
>>> (in a clean install, engine is not aware of VMs running on hosts from
>>> previous installs - we need to cover "import existing storage domain"
>>> in a future version to close this gap. current alternatives around
>>> this are to convert the storage domain to an export domain (for an nfs
>>> data domain) an re-import them to the system or a few other options.
>>>
>>> how many VMs are there?
>> They(VMs) are deleted. So, my situation is this:
>>
>> I have clean install of ovirt 3.3.0beta with imported VMs from export
>> domain. Everything works except snapshots. But when I restart ovirt mgmt
>> server, VMs will disappear. Right now I am doing hourly backup of DB.
>
> this is very fishy...
> can you arrange for a remote access session for someone to take a look?
>
>>
>> I can not use workaround with converting domain to export domain,
>> because we use iscsi. It would be perfect if I can retake existing
>> domain. Another gap is if I dont detach export domain I can not attach
>> it to new datacenter and I have to manually edit some files. It would be
>> absolutely perfect to do it from ovirt web gui (for example by "force
>> override button")
>
> i thought 3.3 allowed for that, but i could be wrong.

Well, I will backup all VMs to export domain, than create snapshot of 
ovirt and I will try to restart that server if its going to delete VMs 
from DB. Than I will report back. I think, that remote session will not 
be possible, but I can be wrong (I hope).