Re: [Users] All VMs disappeared

Cituji "Laszlo Hornyak" <lhornyak(a)redhat.com&gt;: > > > ----- Original Message ----- >> From: "Itamar Heim" <iheim(a)redhat.com&gt; >> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> Cc: "Michal Skrivanek" <mskrivan(a)redhat.com&gt;, users(a)ovirt.org >> Sent: Wednesday, August 14, 2013 4:27:17 PM >> Subject: Re: [Users] All VMs disappeared >> >> On 08/14/2013 05:19 PM, Jakub Bittner wrote: >> > Dne 14.8.2013 10:48, Jakub Bittner napsal(a): >> >> Dne 14.8.2013 09:57, Liron Aravot napsal(a): >> >>> Jakub, are only your vms disappear? what about the disks? >> >>> >> >>> ----- Original Message ----- >> >>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >> >>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> >>>> Cc: users(a)ovirt.org >> >>>> Sent: Wednesday, August 14, 2013 10:39:48 AM >> >>>> Subject: Re: [Users] All VMs disappeared >> >>>> >> >>>> >> >>>> >> >>>> ----- Original Message ----- >> >>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> >>>>> To: users(a)ovirt.org >> >>>>> Sent: Wednesday, August 14, 2013 8:09:45 AM >> >>>>> Subject: Re: [Users] All VMs disappeared >> >>>>> >> >>>>> Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): >> >>>>>> Ahoj Jakub, >> >>>>>> >> >>>>>> Just one more idea: could you turn statement logging on in >> postgresql >> >>>>>> before you try again? >> >>>>>> It is in /var/lib/pgsql/data/postgresql.conf >> >>>>>> log_statement = 'all' >> >>>>>> >> >>>>>> >> >>>>>> ----- Original Message ----- >> >>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> >>>>>>> To: "Itamar Heim" <iheim(a)redhat.com&gt; >> >>>>>>> Cc: users(a)ovirt.org >> >>>>>>> Sent: Tuesday, August 13, 2013 5:20:29 PM >> >>>>>>> Subject: Re: [Users] All VMs disappeared >> >>>>>>> >> >>>>>>> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >> >>>>>>>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >> >>>>>>>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >> >>>>>>>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >> >>>>>>>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >> >>>>>>>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >> >>>>>>>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >> >>>>>>>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >> >>>>>>>>>>>>>>> I looked around Noam's patch and that should not >> cause such >> >>>>>>>>>>>>>>> behavior. >> >>>>>>>>>>>>>>> I am wondering how that lost VM's could happen. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Jakub, can you give a more detailed description what >> you >> >>>>>>>>>>>>>>> were >> >>>>>>>>>>>>>>> doing >> >>>>>>>>>>>>>>> with oVirt when this happened? Maybe the bug is >> still there. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thank you, >> >>>>>>>>>>>>>>> Laszlo >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> ----- Original Message ----- >> >>>>>>>>>>>>>>>> From: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >> >>>>>>>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >> >>>>>>>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam >> Slomianko" >> >>>>>>>>>>>>>>>> <nslomian(a)redhat.com&gt;, users(a)ovirt.org >> >>>>>>>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >> >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >> >>>>>>>>>>>>>>>> NullPointerException. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> Greg >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> ----- Original Message ----- >> >>>>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >> >>>>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt;, "Noam >> Slomianko" >> >>>>>>>>>>>>>>>>> <nslomian(a)redhat.com&gt;, "Greg Sheremeta" >> >>>>>>>>>>>>>>>>> <gshereme(a)redhat.com&gt; >> >>>>>>>>>>>>>>>>> Cc: users(a)ovirt.org >> >>>>>>>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >> >>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Well if they are no longer in DB then that >> explains why >> >>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>> exception no >> >>>>>>>>>>>>>>>>> longer occurs, but at the cost of database >> corruption. >> >>>>>>>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> Thank you, >> >>>>>>>>>>>>>>>>> Laszlo >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> ----- Original Message ----- >> >>>>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> >>>>>>>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme(a)redhat.com&gt; >> >>>>>>>>>>>>>>>>>> Cc: users(a)ovirt.org, "Laszlo Hornyak" >> >>>>>>>>>>>>>>>>>> <lhornyak(a)redhat.com&gt; >> >>>>>>>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >> >>>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >> >>>>>>>>>>>>>>>>>>> It could also be this bug[1], for which I just >> >>>>>>>>>>>>>>>>>>> submitted a >> >>>>>>>>>>>>>>>>>>> fix. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> [1] >> https://bugzilla.redhat.com/show_bug.cgi?id=987907 >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> You can work around it by typing just "VMs:" >> (without >> >>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> quotes) in >> >>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> search bar. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Greg >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> ----- Original Message ----- >> >>>>>>>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt; >> >>>>>>>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> >>>>>>>>>>>>>>>>>>>> Cc: users(a)ovirt.org >> >>>>>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >> >>>>>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Hi Jakub, >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >> >>>>>>>>>>>>>>>>>>>> are in >> >>>>>>>>>>>>>>>>>>>> your >> >>>>>>>>>>>>>>>>>>>> DB? >> >>>>>>>>>>>>>>>>>>>> select * from vm_static; >> >>>>>>>>>>>>>>>>>>>> or >> >>>>>>>>>>>>>>>>>>>> curl -u admin@internal:blablabla >> [engine-url]api/vms >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> It seems Noam fixed this issue already in >> >>>>>>>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Thank you, >> >>>>>>>>>>>>>>>>>>>> Laszlo >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> ----- Original Message ----- >> >>>>>>>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner(a)nbu.cz&gt; >> >>>>>>>>>>>>>>>>>>>>> To: users(a)ovirt.org >> >>>>>>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >> >>>>>>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >> >>>>>>>>>>>>>>>>>>>>>> Hello, >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >> >>>>>>>>>>>>>>>>>>>>>> all our >> >>>>>>>>>>>>>>>>>>>>>> VMs >> >>>>>>>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on >> centos >> >>>>>>>>>>>>>>>>>>>>>> too. >> >>>>>>>>>>>>>>>>>>>>>> Repeating >> >>>>>>>>>>>>>>>>>>>>>> problem in log is: >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >> >>>>>>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >> >>>>>>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >> >>>>>>>>>>>>>>>>>>>>>> refresh VDS >> >>>>>>>>>>>>>>>>>>>>>> , >> >>>>>>>>>>>>>>>>>>>>>> vds = >> >>>>>>>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >> >>>>>>>>>>>>>>>>>>>>>> node2.x.com, error >> >>>>>>>>>>>>>>>>>>>>>> = >> >>>>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >> >>>>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >> >>>>>>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >> >>>>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [scheduler.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> >>>>>>>>>>>>>>>>>>>>>> [quartz.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [quartz.jar:] >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >> >>>>>>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >> >>>>>>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >> >>>>>>>>>>>>>>>>>>>>>> refresh VDS >> >>>>>>>>>>>>>>>>>>>>>> , >> >>>>>>>>>>>>>>>>>>>>>> vds = >> >>>>>>>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >> >>>>>>>>>>>>>>>>>>>>>> node3.x.com, error >> >>>>>>>>>>>>>>>>>>>>>> = >> >>>>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >> >>>>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >> >>>>>>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >> >>>>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [scheduler.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> >>>>>>>>>>>>>>>>>>>>>> [quartz.jar:] >> >>>>>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>> >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> [quartz.jar:] >> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>>>>>>>>>>>>> Users mailing list >> >>>>>>>>>>>>>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>>>>>>>>>>>>> I restarted management node and both nodes and >> this >> >>>>>>>>>>>>>>>>>>>>> problem >> >>>>>>>>>>>>>>>>>>>>> disappeared. >> >>>>>>>>>>>>>>>>>>>>> But I still can not see any VMs. >> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>>>>>>>>>>>> Users mailing list >> >>>>>>>>>>>>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>>>>>>>>>>> Users mailing list >> >>>>>>>>>>>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> I do not have them in database. Both commands shows >> >>>>>>>>>>>>>>>>>> nothing >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hello, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, >> everything was >> >>>>>>>>>>>>>> great >> >>>>>>>>>>>>>> (except SPICE. VNC and creating new WM due to >> serialization >> >>>>>>>>>>>>>> bug.). >> >>>>>>>>>>>>>> Because those bugs were irritating, we switched to >> >>>>>>>>>>>>>> nightly. Still >> >>>>>>>>>>>>>> works fine (but bug with VMs is not fixed there), than I >> >>>>>>>>>>>>>> installed >> >>>>>>>>>>>>>> new >> >>>>>>>>>>>>>> ovirt engine management server in KVM virtualization >> (using >> >>>>>>>>>>>>>> libvirtd) >> >>>>>>>>>>>>>> due to taking snapshots before deploying nightly >> builds. New >> >>>>>>>>>>>>>> virtualized engine works great and I delete old one. >> Next >> >>>>>>>>>>>>>> day I >> >>>>>>>>>>>>>> came >> >>>>>>>>>>>>>> to work and engine web page was very very slow, every >> >>>>>>>>>>>>>> operation >> >>>>>>>>>>>>>> takes >> >>>>>>>>>>>>>> ages, that time I found there were no VMs, no HDDs >> and so. I >> >>>>>>>>>>>>>> converted >> >>>>>>>>>>>>>> its qcow2 image to raw, because I think it has slow DB >> >>>>>>>>>>>>>> because of >> >>>>>>>>>>>>>> running i qcow2. Did not helped, than I take a look >> in nodes >> >>>>>>>>>>>>>> logs and >> >>>>>>>>>>>>>> found this problem, which I posted in first email >> (VMs still >> >>>>>>>>>>>>>> running, >> >>>>>>>>>>>>>> but not visible in engine VMs tab, but on host I can see >> >>>>>>>>>>>>>> number >> >>>>>>>>>>>>>> of >> >>>>>>>>>>>>>> running VMs). After restart bug disappeared, but VMs >> too. >> >>>>>>>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But >> all my >> >>>>>>>>>>>>>> VMs >> >>>>>>>>>>>>>> are >> >>>>>>>>>>>>>> gone. Even from storage. (but disappearing from storage >> >>>>>>>>>>>>>> may be my >> >>>>>>>>>>>>>> fault during recovery). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I can provide logs from both two nodes and >> (hopefully) from >> >>>>>>>>>>>>>> engine >> >>>>>>>>>>>>>> too. I can provide you with those logs, but not >> publicly. >> >>>>>>>>>>>>>> Sorry. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thank you, >> >>>>>>>>>>>>>> Jakub Bittner, NSA Czech >> >>>>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>>>>> Users mailing list >> >>>>>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and >> my VMs >> >>>>>>>>>>>>> are >> >>>>>>>>>>>>> gone >> >>>>>>>>>>>>> again. >> >>>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>>>> Users mailing list >> >>>>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>>>> can you verify the db from before the upgrade has the VMs? >> >>>>>>>>>>> I dont know what you mean by verify database. Please, >> could you >> >>>>>>>>>>> point me >> >>>>>>>>>>> how to do it? Thank you >> >>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>> Users mailing list >> >>>>>>>>>>> Users(a)ovirt.org >> >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>>>>> I assume from your expectation to see the VMs from the >> previous >> >>>>>>>>>> install that you upgraded your system rather than do a clean >> >>>>>>>>>> install? >> >>>>>>>>>> in an upgrade, there should be a backup of the db from >> before the >> >>>>>>>>>> upgrade. >> >>>>>>>>>> (in a clean install, engine is not aware of VMs running on >> >>>>>>>>>> hosts from >> >>>>>>>>>> previous installs - we need to cover "import existing >> storage >> >>>>>>>>>> domain" >> >>>>>>>>>> in a future version to close this gap. current alternatives >> >>>>>>>>>> around >> >>>>>>>>>> this are to convert the storage domain to an export domain >> >>>>>>>>>> (for an >> >>>>>>>>>> nfs >> >>>>>>>>>> data domain) an re-import them to the system or a few other >> >>>>>>>>>> options. >> >>>>>>>>>> >> >>>>>>>>>> how many VMs are there? >> >>>>>>>>> They(VMs) are deleted. So, my situation is this: >> >>>>>>>>> >> >>>>>>>>> I have clean install of ovirt 3.3.0beta with imported VMs >> from >> >>>>>>>>> export >> >>>>>>>>> domain. Everything works except snapshots. But when I restart >> >>>>>>>>> ovirt >> >>>>>>>>> mgmt >> >>>>>>>>> server, VMs will disappear. Right now I am doing hourly >> backup >> >>>>>>>>> of DB. >> >>>>>>>> this is very fishy... >> >>>>>>>> can you arrange for a remote access session for someone to >> take >> >>>>>>>> a look? >> >>>>>>>> >> >>>>>>>>> I can not use workaround with converting domain to export >> domain, >> >>>>>>>>> because we use iscsi. It would be perfect if I can retake >> existing >> >>>>>>>>> domain. Another gap is if I dont detach export domain I >> can not >> >>>>>>>>> attach >> >>>>>>>>> it to new datacenter and I have to manually edit some >> files. It >> >>>>>>>>> would >> >>>>>>>>> be >> >>>>>>>>> absolutely perfect to do it from ovirt web gui (for >> example by >> >>>>>>>>> "force >> >>>>>>>>> override button") >> >>>>>>>> i thought 3.3 allowed for that, but i could be wrong. >> >>>>>>> Well, I will backup all VMs to export domain, than create >> >>>>>>> snapshot of >> >>>>>>> ovirt and I will try to restart that server if its going to >> >>>>>>> delete VMs >> >>>>>>> from DB. Than I will report back. I think, that remote session >> >>>>>>> will not >> >>>>>>> be possible, but I can be wrong (I hope). >> >>>>>>> >> >>>>>>> >> >>>>>>> _______________________________________________ >> >>>>>>> Users mailing list >> >>>>>>> Users(a)ovirt.org >> >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>>>> >> >>>>> I change it. But I can not restart postgres. >> >>>> You can restart it only when the engine is down, it will not >> restart >> >>>> with >> >>>> database connections open. >> >>>> >> >>>>> _______________________________________________ >> >>>>> Users mailing list >> >>>>> Users(a)ovirt.org >> >>>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>>> >> >>>> _______________________________________________ >> >>>> Users mailing list >> >>>> Users(a)ovirt.org >> >>>> http://lists.ovirt.org/mailman/listinfo/users >> >>>> >> >> I think, that disks did not disappeared, but new instance of ovirt >> >> deleted them as I thought that it can import them back. >> >> _______________________________________________ >> >> Users mailing list >> >> Users(a)ovirt.org >> >> http://lists.ovirt.org/mailman/listinfo/users >> > >> > >> > hello, >> > >> > right now I restarted ovirt-engine and postgresql and my VMs >> disappeared >> > again, but not all of them! Some of them persists. (two) And I get >> > messages about failed export and import of those who disappeared. >> Than I >> > restored postgresql from backup and everything is OK. Restarting >> > postgresql is like using way back machine ;-) >> >> Laszlo hinted this may be around some compnesation logic - detecting an >> incomplete import, thus deleting them. yair - what should the logs >> indicate for this? > > I thought of the compensation because it runs after engine start and > does changes in the DB if there is something to compensate in > business_entity_snapshot. > So for example having records there could undo the import of VM's and > at the same time delete their disks from the storage. > However, when last time I had anything to do with the compensation, > if it failed, it did not allow the engine to start. I don't know if > it is still like that, I just noticed last week that some services > may fail to configure and the engine ignores them in such cases. > > Jakub, do you have anything in the business_entity_snapshot table? > >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > I do appologies, but I will be at work on Friday 23 and than I take a look. Sorry. But thank you for hints ;-) ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users