Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
Dne 12.8.2013 22:44, Itamar Heim napsal(a): > On 08/12/2013 06:59 PM, Jakub Bittner wrote: >> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>> I looked around Noam's patch and that should not cause such >>>> behavior. >>>> I am wondering how that lost VM's could happen. >>>> >>>> Jakub, can you give a more detailed description what you were >>>> doing >>>> with oVirt when this happened? Maybe the bug is still there. >>>> >>>> Thank you, >>>> Laszlo >>>> >>>> ----- Original Message ----- >>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>> <nslomian@redhat.com>, users@ovirt.org >>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Not the one I fixed, 987907. It was a simple UI >>>>> NullPointerException. >>>>> >>>>> Greg >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>> <gshereme@redhat.com> >>>>>> Cc: users@ovirt.org >>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Well if they are no longer in DB then that explains why the >>>>>> exception no >>>>>> longer occurs, but at the cost of database corruption. >>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>> >>>>>> Thank you, >>>>>> Laszlo >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>> fix. >>>>>>>> >>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>> >>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>> quotes) in >>>>>>>> the >>>>>>>> search bar. >>>>>>>> >>>>>>>> Greg >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>> Cc: users@ovirt.org >>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Hi Jakub, >>>>>>>>> >>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>> your >>>>>>>>> DB? >>>>>>>>> select * from vm_static; >>>>>>>>> or >>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>> >>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Laszlo >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>> To: users@ovirt.org >>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>> Repeating >>>>>>>>>>> problem in log is: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>>> vds = >>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> >>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>>> vds = >>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>>> disappeared. >>>>>>>>>> But I still can not see any VMs. >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I do not have them in database. Both commands shows nothing >>>>>>> >>> Hello, >>> >>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >>> (except SPICE. VNC and creating new WM due to serialization bug.). >>> Because those bugs were irritating, we switched to nightly. Still >>> works fine (but bug with VMs is not fixed there), than I installed >>> new >>> ovirt engine management server in KVM virtualization (using >>> libvirtd) >>> due to taking snapshots before deploying nightly builds. New >>> virtualized engine works great and I delete old one. Next day I >>> came >>> to work and engine web page was very very slow, every operation >>> takes >>> ages, that time I found there were no VMs, no HDDs and so. I >>> converted >>> its qcow2 image to raw, because I think it has slow DB because of >>> running i qcow2. Did not helped, than I take a look in nodes >>> logs and >>> found this problem, which I posted in first email (VMs still >>> running, >>> but not visible in engine VMs tab, but on host I can see number of >>> running VMs). After restart bug disappeared, but VMs too. >>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >>> gone. Even from storage. (but disappearing from storage may be my >>> fault during recovery). >>> >>> I can provide logs from both two nodes and (hopefully) from engine >>> too. I can provide you with those logs, but not publicly. Sorry. >>> >>> Thank you, >>> Jakub Bittner, NSA Czech >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone >> again. >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > can you verify the db from before the upgrade has the VMs? I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from
On 08/13/2013 08:50 AM, Jakub Bittner wrote: previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
On 08/13/2013 05:31 PM, Jakub Bittner wrote: this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button") i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres.