Re: [Users] All VMs disappeared

On 08/13/2013 08:50 AM, Jakub Bittner wrote: > Dne 12.8.2013 22:44, Itamar Heim napsal(a): >> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>> I looked around Noam's patch and that should not cause such >>>>> behavior. >>>>> I am wondering how that lost VM's could happen. >>>>> >>>>> Jakub, can you give a more detailed description what you were doing >>>>> with oVirt when this happened? Maybe the bug is still there. >>>>> >>>>> Thank you, >>>>> Laszlo >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>> To: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>> Cc: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>>> <nslomian(a)redhat.com&gt;, users(a)ovirt.org >>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>> NullPointerException. >>>>>> >>>>>> Greg >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>>>> <nslomian(a)redhat.com&gt;, "Greg Sheremeta" >>>>>>> <gshereme(a)redhat.com&gt; >>>>>>> Cc: users(a)ovirt.org >>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Well if they are no longer in DB then that explains why the >>>>>>> exception no >>>>>>> longer occurs, but at the cost of database corruption. >>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>> >>>>>>> Thank you, >>>>>>> Laszlo >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>> To: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>>>> Cc: users(a)ovirt.org, "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>> It could also be this bug[1], for which I just submitted a fix. >>>>>>>>> >>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>> >>>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>>> quotes) in >>>>>>>>> the >>>>>>>>> search bar. >>>>>>>>> >>>>>>>>> Greg >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Hi Jakub, >>>>>>>>>> >>>>>>>>>> Could you check through DB or REST-API if the VM's are in your >>>>>>>>>> DB? >>>>>>>>>> select * from vm_static; >>>>>>>>>> or >>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>> >>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>>> To: users(a)ovirt.org >>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>>> Repeating >>>>>>>>>>>> problem in log is: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>>>> vds = >>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> >>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>>>> vds = >>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>>>> disappeared. >>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I do not have them in database. Both commands shows nothing >>>>>>>> >>>> Hello, >>>> >>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >>>> (except SPICE. VNC and creating new WM due to serialization bug.). >>>> Because those bugs were irritating, we switched to nightly. Still >>>> works fine (but bug with VMs is not fixed there), than I installed >>>> new >>>> ovirt engine management server in KVM virtualization (using libvirtd) >>>> due to taking snapshots before deploying nightly builds. New >>>> virtualized engine works great and I delete old one. Next day I came >>>> to work and engine web page was very very slow, every operation takes >>>> ages, that time I found there were no VMs, no HDDs and so. I >>>> converted >>>> its qcow2 image to raw, because I think it has slow DB because of >>>> running i qcow2. Did not helped, than I take a look in nodes logs and >>>> found this problem, which I posted in first email (VMs still running, >>>> but not visible in engine VMs tab, but on host I can see number of >>>> running VMs). After restart bug disappeared, but VMs too. >>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >>>> gone. Even from storage. (but disappearing from storage may be my >>>> fault during recovery). >>>> >>>> I can provide logs from both two nodes and (hopefully) from engine >>>> too. I can provide you with those logs, but not publicly. Sorry. >>>> >>>> Thank you, >>>> Jakub Bittner, NSA Czech >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone >>> again. >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> can you verify the db from before the upgrade has the VMs? > > I dont know what you mean by verify database. Please, could you point me > how to do it? Thank you > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options. how many VMs are there?