Re: [Users] All VMs disappeared

On 08/13/2013 05:31 PM, Jakub Bittner wrote: > Dne 13.8.2013 15:39, Itamar Heim napsal(a): >> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>> I looked around Noam's patch and that should not cause such >>>>>>> behavior. >>>>>>> I am wondering how that lost VM's could happen. >>>>>>> >>>>>>> Jakub, can you give a more detailed description what you were >>>>>>> doing >>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>> >>>>>>> Thank you, >>>>>>> Laszlo >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>>>> To: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>> Cc: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>>>>> <nslomian(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>> NullPointerException. >>>>>>>> >>>>>>>> Greg >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>>>>>> <nslomian(a)redhat.com&gt;, "Greg Sheremeta" >>>>>>>>> <gshereme(a)redhat.com&gt; >>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Well if they are no longer in DB then that explains why the >>>>>>>>> exception no >>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Laszlo >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>> To: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>>>>>> Cc: users(a)ovirt.org, "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>>>>> fix. >>>>>>>>>>> >>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>> >>>>>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>>>>> quotes) in >>>>>>>>>>> the >>>>>>>>>>> search bar. >>>>>>>>>>> >>>>>>>>>>> Greg >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>> >>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>>>>> your >>>>>>>>>>>> DB? >>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>> or >>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>> >>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Laszlo >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>>>>> To: users(a)ovirt.org >>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>>>>>> vds = >>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>>>>>> vds = >>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>>>>>> disappeared. >>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I do not have them in database. Both commands shows nothing >>>>>>>>>> >>>>>> Hello, >>>>>> >>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >>>>>> (except SPICE. VNC and creating new WM due to serialization bug.). >>>>>> Because those bugs were irritating, we switched to nightly. Still >>>>>> works fine (but bug with VMs is not fixed there), than I installed >>>>>> new >>>>>> ovirt engine management server in KVM virtualization (using >>>>>> libvirtd) >>>>>> due to taking snapshots before deploying nightly builds. New >>>>>> virtualized engine works great and I delete old one. Next day I >>>>>> came >>>>>> to work and engine web page was very very slow, every operation >>>>>> takes >>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>> converted >>>>>> its qcow2 image to raw, because I think it has slow DB because of >>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>> logs and >>>>>> found this problem, which I posted in first email (VMs still >>>>>> running, >>>>>> but not visible in engine VMs tab, but on host I can see number of >>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >>>>>> gone. Even from storage. (but disappearing from storage may be my >>>>>> fault during recovery). >>>>>> >>>>>> I can provide logs from both two nodes and (hopefully) from engine >>>>>> too. I can provide you with those logs, but not publicly. Sorry. >>>>>> >>>>>> Thank you, >>>>>> Jakub Bittner, NSA Czech >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone >>>>> again. >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> can you verify the db from before the upgrade has the VMs? >>> >>> I dont know what you mean by verify database. Please, could you >>> point me >>> how to do it? Thank you >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> I assume from your expectation to see the VMs from the previous >> install that you upgraded your system rather than do a clean install? >> in an upgrade, there should be a backup of the db from before the >> upgrade. >> (in a clean install, engine is not aware of VMs running on hosts from >> previous installs - we need to cover "import existing storage domain" >> in a future version to close this gap. current alternatives around >> this are to convert the storage domain to an export domain (for an nfs >> data domain) an re-import them to the system or a few other options. >> >> how many VMs are there? > They(VMs) are deleted. So, my situation is this: > > I have clean install of ovirt 3.3.0beta with imported VMs from export > domain. Everything works except snapshots. But when I restart ovirt mgmt > server, VMs will disappear. Right now I am doing hourly backup of DB. this is very fishy... can you arrange for a remote access session for someone to take a look? > > I can not use workaround with converting domain to export domain, > because we use iscsi. It would be perfect if I can retake existing > domain. Another gap is if I dont detach export domain I can not attach > it to new datacenter and I have to manually edit some files. It would be > absolutely perfect to do it from ovirt web gui (for example by "force > override button") i thought 3.3 allowed for that, but i could be wrong.