Re: [Users] All VMs disappeared

From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; To: users(a)ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): > Ahoj Jakub, > > Just one more idea: could you turn statement logging on in postgresql > before you try again? > It is in /var/lib/pgsql/data/postgresql.conf > log_statement = 'all' > > > ----- Original Message ----- >> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> To: "Itamar Heim" <iheim(a)redhat.com&gt; >> Cc: users(a)ovirt.org >> Sent: Tuesday, August 13, 2013 5:20:29 PM >> Subject: Re: [Users] All VMs disappeared >> >> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>>> behavior. >>>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>>> >>>>>>>>>> Jakub, can you give a more detailed description what you were >>>>>>>>>> doing >>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>>>>>>>> <nslomian(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>>> NullPointerException. >>>>>>>>>>> >>>>>>>>>>> Greg >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam Slomianko" >>>>>>>>>>>> <nslomian(a)redhat.com&gt;, "Greg Sheremeta" >>>>>>>>>>>> <gshereme(a)redhat.com&gt; >>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Well if they are no longer in DB then that explains why the >>>>>>>>>>>> exception no >>>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Laszlo >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >>>>>>>>>>>>> Cc: users(a)ovirt.org, "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>>>>>>>> fix. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>>>>>>>> quotes) in >>>>>>>>>>>>>> the >>>>>>>>>>>>>> search bar. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Greg >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> DB? >>>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >>>>>>>>>>>>>>>> To: users(a)ovirt.org >>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> Users(a)ovirt.org >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I do not have them in database. Both commands shows nothing >>>>>>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >>>>>>>>> (except SPICE. VNC and creating new WM due to serialization bug.). >>>>>>>>> Because those bugs were irritating, we switched to nightly. Still >>>>>>>>> works fine (but bug with VMs is not fixed there), than I installed >>>>>>>>> new >>>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>>> libvirtd) >>>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>>> virtualized engine works great and I delete old one. Next day I >>>>>>>>> came >>>>>>>>> to work and engine web page was very very slow, every operation >>>>>>>>> takes >>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>>> converted >>>>>>>>> its qcow2 image to raw, because I think it has slow DB because of >>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>>> logs and >>>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>>> running, >>>>>>>>> but not visible in engine VMs tab, but on host I can see number of >>>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >>>>>>>>> gone. Even from storage. (but disappearing from storage may be my >>>>>>>>> fault during recovery). >>>>>>>>> >>>>>>>>> I can provide logs from both two nodes and (hopefully) from engine >>>>>>>>> too. I can provide you with those logs, but not publicly. Sorry. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Jakub Bittner, NSA Czech >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone >>>>>>>> again. >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> can you verify the db from before the upgrade has the VMs? >>>>>> I dont know what you mean by verify database. Please, could you >>>>>> point me >>>>>> how to do it? Thank you >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> I assume from your expectation to see the VMs from the previous >>>>> install that you upgraded your system rather than do a clean install? >>>>> in an upgrade, there should be a backup of the db from before the >>>>> upgrade. >>>>> (in a clean install, engine is not aware of VMs running on hosts from >>>>> previous installs - we need to cover "import existing storage domain" >>>>> in a future version to close this gap. current alternatives around >>>>> this are to convert the storage domain to an export domain (for an nfs >>>>> data domain) an re-import them to the system or a few other options. >>>>> >>>>> how many VMs are there? >>>> They(VMs) are deleted. So, my situation is this: >>>> >>>> I have clean install of ovirt 3.3.0beta with imported VMs from export >>>> domain. Everything works except snapshots. But when I restart ovirt mgmt >>>> server, VMs will disappear. Right now I am doing hourly backup of DB. >>> this is very fishy... >>> can you arrange for a remote access session for someone to take a look? >>> >>>> I can not use workaround with converting domain to export domain, >>>> because we use iscsi. It would be perfect if I can retake existing >>>> domain. Another gap is if I dont detach export domain I can not attach >>>> it to new datacenter and I have to manually edit some files. It would be >>>> absolutely perfect to do it from ovirt web gui (for example by "force >>>> override button") >>> i thought 3.3 allowed for that, but i could be wrong. >> >> Well, I will backup all VMs to export domain, than create snapshot of >> ovirt and I will try to restart that server if its going to delete VMs >> from DB. Than I will report back. I think, that remote session will not >> be possible, but I can be wrong (I hope). >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> I change it. But I can not restart postgres.

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users