Re: [Users] All VMs disappeared

On 08/12/2013 06:59 PM, Jakub Bittner wrote: > Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>> I looked around Noam's patch and that should not cause such behavior. >>> I am wondering how that lost VM's could happen. >>> >>> Jakub, can you give a more detailed description what you were doing >>> with oVirt when this happened? Maybe the bug is still there. >>> >>> Thank you, >>> Laszlo >>> >>> ----- Original Message ----- >>>> From: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>> To: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>> Cc: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>> <nslomian(a)redhat.com&gt;, users(a)ovirt.org >>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Not the one I fixed, 987907. It was a simple UI NullPointerException. >>>> >>>> Greg >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>> <nslomian(a)redhat.com&gt;, "Greg Sheremeta" >>>>> <gshereme(a)redhat.com&gt; >>>>> Cc: users(a)ovirt.org >>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Well if they are no longer in DB then that explains why the >>>>> exception no >>>>> longer occurs, but at the cost of database corruption. >>>>> Noam and Greg, can these bugs cause data corruption? >>>>> >>>>> Thank you, >>>>> Laszlo >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>> To: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>> Cc: users(a)ovirt.org, "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>> It could also be this bug[1], for which I just submitted a fix. >>>>>>> >>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>> >>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>> quotes) in >>>>>>> the >>>>>>> search bar. >>>>>>> >>>>>>> Greg >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>> Cc: users(a)ovirt.org >>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Hi Jakub, >>>>>>>> >>>>>>>> Could you check through DB or REST-API if the VM's are in your >>>>>>>> DB? >>>>>>>> select * from vm_static; >>>>>>>> or >>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>> >>>>>>>> It seems Noam fixed this issue already in >>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Laszlo >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>> To: users(a)ovirt.org >>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>> Repeating >>>>>>>>>> problem in log is: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>> vds = >>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [scheduler.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>> [quartz.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [quartz.jar:] >>>>>>>>>> >>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>> vds = >>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [scheduler.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>> [quartz.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [quartz.jar:] >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>> disappeared. >>>>>>>>> But I still can not see any VMs. >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>> Hi, >>>>>> >>>>>> I do not have them in database. Both commands shows nothing >>>>>> >> Hello, >> >> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >> (except SPICE. VNC and creating new WM due to serialization bug.). >> Because those bugs were irritating, we switched to nightly. Still >> works fine (but bug with VMs is not fixed there), than I installed new >> ovirt engine management server in KVM virtualization (using libvirtd) >> due to taking snapshots before deploying nightly builds. New >> virtualized engine works great and I delete old one. Next day I came >> to work and engine web page was very very slow, every operation takes >> ages, that time I found there were no VMs, no HDDs and so. I converted >> its qcow2 image to raw, because I think it has slow DB because of >> running i qcow2. Did not helped, than I take a look in nodes logs and >> found this problem, which I posted in first email (VMs still running, >> but not visible in engine VMs tab, but on host I can see number of >> running VMs). After restart bug disappeared, but VMs too. >> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >> gone. Even from storage. (but disappearing from storage may be my >> fault during recovery). >> >> I can provide logs from both two nodes and (hopefully) from engine >> too. I can provide you with those logs, but not publicly. Sorry. >> >> Thank you, >> Jakub Bittner, NSA Czech >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone > again. > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users can you verify the db from before the upgrade has the VMs?