[Users] All VMs disappeared
Jakub Bittner
j.bittner at nbu.cz
Tue Aug 13 14:31:36 UTC 2013
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
> On 08/13/2013 08:50 AM, Jakub Bittner wrote:
>> Dne 12.8.2013 22:44, Itamar Heim napsal(a):
>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote:
>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
>>>>>> I looked around Noam's patch and that should not cause such
>>>>>> behavior.
>>>>>> I am wondering how that lost VM's could happen.
>>>>>>
>>>>>> Jakub, can you give a more detailed description what you were doing
>>>>>> with oVirt when this happened? Maybe the bug is still there.
>>>>>>
>>>>>> Thank you,
>>>>>> Laszlo
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>> Cc: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>> <nslomian at redhat.com>, users at ovirt.org
>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM
>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>
>>>>>>> Not the one I fixed, 987907. It was a simple UI
>>>>>>> NullPointerException.
>>>>>>>
>>>>>>> Greg
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>>> <nslomian at redhat.com>, "Greg Sheremeta"
>>>>>>>> <gshereme at redhat.com>
>>>>>>>> Cc: users at ovirt.org
>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM
>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>
>>>>>>>> Well if they are no longer in DB then that explains why the
>>>>>>>> exception no
>>>>>>>> longer occurs, but at the cost of database corruption.
>>>>>>>> Noam and Greg, can these bugs cause data corruption?
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>> Laszlo
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>> To: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>>>> Cc: users at ovirt.org, "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>
>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
>>>>>>>>>> It could also be this bug[1], for which I just submitted a fix.
>>>>>>>>>>
>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
>>>>>>>>>>
>>>>>>>>>> You can work around it by typing just "VMs:" (without the
>>>>>>>>>> quotes) in
>>>>>>>>>> the
>>>>>>>>>> search bar.
>>>>>>>>>>
>>>>>>>>>> Greg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>
>>>>>>>>>>> Hi Jakub,
>>>>>>>>>>>
>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in your
>>>>>>>>>>> DB?
>>>>>>>>>>> select * from vm_static;
>>>>>>>>>>> or
>>>>>>>>>>> curl -u admin at internal:blablabla [engine-url]api/vms
>>>>>>>>>>>
>>>>>>>>>>> It seems Noam fixed this issue already in
>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
>>>>>>>>>>>
>>>>>>>>>>> Thank you,
>>>>>>>>>>> Laszlo
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>>> To: users at ovirt.org
>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>
>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs
>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too.
>>>>>>>>>>>>> Repeating
>>>>>>>>>>>>> problem in log is:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN
>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS ,
>>>>>>>>>>>>> vds =
>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error =
>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN
>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS ,
>>>>>>>>>>>>> vds =
>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error =
>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>> I restarted management node and both nodes and this problem
>>>>>>>>>>>> disappeared.
>>>>>>>>>>>> But I still can not see any VMs.
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I do not have them in database. Both commands shows nothing
>>>>>>>>>
>>>>> Hello,
>>>>>
>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great
>>>>> (except SPICE. VNC and creating new WM due to serialization bug.).
>>>>> Because those bugs were irritating, we switched to nightly. Still
>>>>> works fine (but bug with VMs is not fixed there), than I installed
>>>>> new
>>>>> ovirt engine management server in KVM virtualization (using libvirtd)
>>>>> due to taking snapshots before deploying nightly builds. New
>>>>> virtualized engine works great and I delete old one. Next day I came
>>>>> to work and engine web page was very very slow, every operation takes
>>>>> ages, that time I found there were no VMs, no HDDs and so. I
>>>>> converted
>>>>> its qcow2 image to raw, because I think it has slow DB because of
>>>>> running i qcow2. Did not helped, than I take a look in nodes logs and
>>>>> found this problem, which I posted in first email (VMs still running,
>>>>> but not visible in engine VMs tab, but on host I can see number of
>>>>> running VMs). After restart bug disappeared, but VMs too.
>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are
>>>>> gone. Even from storage. (but disappearing from storage may be my
>>>>> fault during recovery).
>>>>>
>>>>> I can provide logs from both two nodes and (hopefully) from engine
>>>>> too. I can provide you with those logs, but not publicly. Sorry.
>>>>>
>>>>> Thank you,
>>>>> Jakub Bittner, NSA Czech
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone
>>>> again.
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>> can you verify the db from before the upgrade has the VMs?
>>
>> I dont know what you mean by verify database. Please, could you point me
>> how to do it? Thank you
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> I assume from your expectation to see the VMs from the previous
> install that you upgraded your system rather than do a clean install?
> in an upgrade, there should be a backup of the db from before the
> upgrade.
> (in a clean install, engine is not aware of VMs running on hosts from
> previous installs - we need to cover "import existing storage domain"
> in a future version to close this gap. current alternatives around
> this are to convert the storage domain to an export domain (for an nfs
> data domain) an re-import them to the system or a few other options.
>
> how many VMs are there?
They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export
domain. Everything works except snapshots. But when I restart ovirt mgmt
server, VMs will disappear. Right now I am doing hourly backup of DB.
I can not use workaround with converting domain to export domain,
because we use iscsi. It would be perfect if I can retake existing
domain. Another gap is if I dont detach export domain I can not attach
it to new datacenter and I have to manually edit some files. It would be
absolutely perfect to do it from ovirt web gui (for example by "force
override button")
More information about the Users
mailing list