Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a): > On 08/13/2013 05:31 PM, Jakub Bittner wrote: >> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>> behavior. >>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>> >>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>> were >>>>>>>> doing >>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Laszlo >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>> NullPointerException. >>>>>>>>> >>>>>>>>> Greg >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>> the >>>>>>>>>> exception no >>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>> submitted a >>>>>>>>>>>> fix. >>>>>>>>>>>> >>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>> >>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>> the >>>>>>>>>>>> quotes) in >>>>>>>>>>>> the >>>>>>>>>>>> search bar. >>>>>>>>>>>> >>>>>>>>>>>> Greg >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>> >>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>> are in >>>>>>>>>>>>> your >>>>>>>>>>>>> DB? >>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>> or >>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>> >>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> Laszlo >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>> , >>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>> = >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>> , >>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>> = >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>> problem >>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>> nothing >>>>>>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>> great >>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>> bug.). >>>>>>> Because those bugs were irritating, we switched to >>>>>>> nightly. Still >>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>> installed >>>>>>> new >>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>> libvirtd) >>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>> virtualized engine works great and I delete old one. Next >>>>>>> day I >>>>>>> came >>>>>>> to work and engine web page was very very slow, every >>>>>>> operation >>>>>>> takes >>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>> converted >>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>> because of >>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>> logs and >>>>>>> found this problem, which I posted in first email (VMs still >>>>>>> running, >>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>> number >>>>>>> of >>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>> VMs >>>>>>> are >>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>> may be my >>>>>>> fault during recovery). >>>>>>> >>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>> engine >>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>> Sorry. >>>>>>> >>>>>>> Thank you, >>>>>>> Jakub Bittner, NSA Czech >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>> are >>>>>> gone >>>>>> again. >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> can you verify the db from before the upgrade has the VMs? >>>> I dont know what you mean by verify database. Please, could you >>>> point me >>>> how to do it? Thank you >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> I assume from your expectation to see the VMs from the previous >>> install that you upgraded your system rather than do a clean >>> install? >>> in an upgrade, there should be a backup of the db from before the >>> upgrade. >>> (in a clean install, engine is not aware of VMs running on >>> hosts from >>> previous installs - we need to cover "import existing storage >>> domain" >>> in a future version to close this gap. current alternatives >>> around >>> this are to convert the storage domain to an export domain >>> (for an >>> nfs >>> data domain) an re-import them to the system or a few other >>> options. >>> >>> how many VMs are there? >> They(VMs) are deleted. So, my situation is this: >> >> I have clean install of ovirt 3.3.0beta with imported VMs from >> export >> domain. Everything works except snapshots. But when I restart >> ovirt >> mgmt >> server, VMs will disappear. Right now I am doing hourly backup >> of DB. > this is very fishy... > can you arrange for a remote access session for someone to take > a look? > >> I can not use workaround with converting domain to export domain, >> because we use iscsi. It would be perfect if I can retake existing >> domain. Another gap is if I dont detach export domain I can not >> attach >> it to new datacenter and I have to manually edit some files. It >> would >> be >> absolutely perfect to do it from ovirt web gui (for example by >> "force >> override button") > i thought 3.3 allowed for that, but i could be wrong. Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello, right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)