[Users] All VMs disappeared
Laszlo Hornyak
lhornyak at redhat.com
Tue Aug 13 16:29:53 UTC 2013
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again?
It is in /var/lib/pgsql/data/postgresql.conf
log_statement = 'all'
----- Original Message -----
> From: "Jakub Bittner" <j.bittner at nbu.cz>
> To: "Itamar Heim" <iheim at redhat.com>
> Cc: users at ovirt.org
> Sent: Tuesday, August 13, 2013 5:20:29 PM
> Subject: Re: [Users] All VMs disappeared
>
> Dne 13.8.2013 17:02, Itamar Heim napsal(a):
> > On 08/13/2013 05:31 PM, Jakub Bittner wrote:
> >> Dne 13.8.2013 15:39, Itamar Heim napsal(a):
> >>> On 08/13/2013 08:50 AM, Jakub Bittner wrote:
> >>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a):
> >>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote:
> >>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
> >>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
> >>>>>>>> I looked around Noam's patch and that should not cause such
> >>>>>>>> behavior.
> >>>>>>>> I am wondering how that lost VM's could happen.
> >>>>>>>>
> >>>>>>>> Jakub, can you give a more detailed description what you were
> >>>>>>>> doing
> >>>>>>>> with oVirt when this happened? Maybe the bug is still there.
> >>>>>>>>
> >>>>>>>> Thank you,
> >>>>>>>> Laszlo
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> From: "Greg Sheremeta" <gshereme at redhat.com>
> >>>>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>>>>>>> Cc: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
> >>>>>>>>> <nslomian at redhat.com>, users at ovirt.org
> >>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM
> >>>>>>>>> Subject: Re: [Users] All VMs disappeared
> >>>>>>>>>
> >>>>>>>>> Not the one I fixed, 987907. It was a simple UI
> >>>>>>>>> NullPointerException.
> >>>>>>>>>
> >>>>>>>>> Greg
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ----- Original Message -----
> >>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
> >>>>>>>>>> <nslomian at redhat.com>, "Greg Sheremeta"
> >>>>>>>>>> <gshereme at redhat.com>
> >>>>>>>>>> Cc: users at ovirt.org
> >>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM
> >>>>>>>>>> Subject: Re: [Users] All VMs disappeared
> >>>>>>>>>>
> >>>>>>>>>> Well if they are no longer in DB then that explains why the
> >>>>>>>>>> exception no
> >>>>>>>>>> longer occurs, but at the cost of database corruption.
> >>>>>>>>>> Noam and Greg, can these bugs cause data corruption?
> >>>>>>>>>>
> >>>>>>>>>> Thank you,
> >>>>>>>>>> Laszlo
> >>>>>>>>>>
> >>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
> >>>>>>>>>>> To: "Greg Sheremeta" <gshereme at redhat.com>
> >>>>>>>>>>> Cc: users at ovirt.org, "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
> >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
> >>>>>>>>>>>
> >>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
> >>>>>>>>>>>> It could also be this bug[1], for which I just submitted a
> >>>>>>>>>>>> fix.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
> >>>>>>>>>>>>
> >>>>>>>>>>>> You can work around it by typing just "VMs:" (without the
> >>>>>>>>>>>> quotes) in
> >>>>>>>>>>>> the
> >>>>>>>>>>>> search bar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Greg
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
> >>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>
> >>>>>>>>>>>>> Cc: users at ovirt.org
> >>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
> >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Jakub,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in
> >>>>>>>>>>>>> your
> >>>>>>>>>>>>> DB?
> >>>>>>>>>>>>> select * from vm_static;
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>> curl -u admin at internal:blablabla [engine-url]api/vms
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It seems Noam fixed this issue already in
> >>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>> Laszlo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
> >>>>>>>>>>>>>> To: users at ovirt.org
> >>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
> >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
> >>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs
> >>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too.
> >>>>>>>>>>>>>>> Repeating
> >>>>>>>>>>>>>>> problem in log is:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN
> >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
> >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS ,
> >>>>>>>>>>>>>>> vds =
> >>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error =
> >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
> >>>>>>>>>>>>>>> java.lang.NullPointerException
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
> >>>>>>>>>>>>>>> Source) [:1.7.0_25]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [scheduler.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> >>>>>>>>>>>>>>> [quartz.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [quartz.jar:]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN
> >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
> >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS ,
> >>>>>>>>>>>>>>> vds =
> >>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error =
> >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
> >>>>>>>>>>>>>>> java.lang.NullPointerException
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [vdsbroker.jar:]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
> >>>>>>>>>>>>>>> Source) [:1.7.0_25]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [scheduler.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> >>>>>>>>>>>>>>> [quartz.jar:]
> >>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [quartz.jar:]
> >>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>> Users mailing list
> >>>>>>>>>>>>>>> Users at ovirt.org
> >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>>>>>>>>>>> I restarted management node and both nodes and this problem
> >>>>>>>>>>>>>> disappeared.
> >>>>>>>>>>>>>> But I still can not see any VMs.
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> Users mailing list
> >>>>>>>>>>>>>> Users at ovirt.org
> >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> Users mailing list
> >>>>>>>>>>>>> Users at ovirt.org
> >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I do not have them in database. Both commands shows nothing
> >>>>>>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great
> >>>>>>> (except SPICE. VNC and creating new WM due to serialization bug.).
> >>>>>>> Because those bugs were irritating, we switched to nightly. Still
> >>>>>>> works fine (but bug with VMs is not fixed there), than I installed
> >>>>>>> new
> >>>>>>> ovirt engine management server in KVM virtualization (using
> >>>>>>> libvirtd)
> >>>>>>> due to taking snapshots before deploying nightly builds. New
> >>>>>>> virtualized engine works great and I delete old one. Next day I
> >>>>>>> came
> >>>>>>> to work and engine web page was very very slow, every operation
> >>>>>>> takes
> >>>>>>> ages, that time I found there were no VMs, no HDDs and so. I
> >>>>>>> converted
> >>>>>>> its qcow2 image to raw, because I think it has slow DB because of
> >>>>>>> running i qcow2. Did not helped, than I take a look in nodes
> >>>>>>> logs and
> >>>>>>> found this problem, which I posted in first email (VMs still
> >>>>>>> running,
> >>>>>>> but not visible in engine VMs tab, but on host I can see number of
> >>>>>>> running VMs). After restart bug disappeared, but VMs too.
> >>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are
> >>>>>>> gone. Even from storage. (but disappearing from storage may be my
> >>>>>>> fault during recovery).
> >>>>>>>
> >>>>>>> I can provide logs from both two nodes and (hopefully) from engine
> >>>>>>> too. I can provide you with those logs, but not publicly. Sorry.
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> Jakub Bittner, NSA Czech
> >>>>>>> _______________________________________________
> >>>>>>> Users mailing list
> >>>>>>> Users at ovirt.org
> >>>>>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>>>
> >>>>>>
> >>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone
> >>>>>> again.
> >>>>>> _______________________________________________
> >>>>>> Users mailing list
> >>>>>> Users at ovirt.org
> >>>>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>>
> >>>>> can you verify the db from before the upgrade has the VMs?
> >>>>
> >>>> I dont know what you mean by verify database. Please, could you
> >>>> point me
> >>>> how to do it? Thank you
> >>>> _______________________________________________
> >>>> Users mailing list
> >>>> Users at ovirt.org
> >>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>
> >>> I assume from your expectation to see the VMs from the previous
> >>> install that you upgraded your system rather than do a clean install?
> >>> in an upgrade, there should be a backup of the db from before the
> >>> upgrade.
> >>> (in a clean install, engine is not aware of VMs running on hosts from
> >>> previous installs - we need to cover "import existing storage domain"
> >>> in a future version to close this gap. current alternatives around
> >>> this are to convert the storage domain to an export domain (for an nfs
> >>> data domain) an re-import them to the system or a few other options.
> >>>
> >>> how many VMs are there?
> >> They(VMs) are deleted. So, my situation is this:
> >>
> >> I have clean install of ovirt 3.3.0beta with imported VMs from export
> >> domain. Everything works except snapshots. But when I restart ovirt mgmt
> >> server, VMs will disappear. Right now I am doing hourly backup of DB.
> >
> > this is very fishy...
> > can you arrange for a remote access session for someone to take a look?
> >
> >>
> >> I can not use workaround with converting domain to export domain,
> >> because we use iscsi. It would be perfect if I can retake existing
> >> domain. Another gap is if I dont detach export domain I can not attach
> >> it to new datacenter and I have to manually edit some files. It would be
> >> absolutely perfect to do it from ovirt web gui (for example by "force
> >> override button")
> >
> > i thought 3.3 allowed for that, but i could be wrong.
>
>
> Well, I will backup all VMs to export domain, than create snapshot of
> ovirt and I will try to restart that server if its going to delete VMs
> from DB. Than I will report back. I think, that remote session will not
> be possible, but I can be wrong (I hope).
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
More information about the Users
mailing list