[Users] All VMs disappeared
Jakub Bittner
j.bittner at nbu.cz
Wed Aug 14 06:09:45 UTC 2013
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
> Ahoj Jakub,
>
> Just one more idea: could you turn statement logging on in postgresql before you try again?
> It is in /var/lib/pgsql/data/postgresql.conf
> log_statement = 'all'
>
>
> ----- Original Message -----
>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>> To: "Itamar Heim" <iheim at redhat.com>
>> Cc: users at ovirt.org
>> Sent: Tuesday, August 13, 2013 5:20:29 PM
>> Subject: Re: [Users] All VMs disappeared
>>
>> Dne 13.8.2013 17:02, Itamar Heim napsal(a):
>>> On 08/13/2013 05:31 PM, Jakub Bittner wrote:
>>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a):
>>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote:
>>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a):
>>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote:
>>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
>>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
>>>>>>>>>> I looked around Noam's patch and that should not cause such
>>>>>>>>>> behavior.
>>>>>>>>>> I am wondering how that lost VM's could happen.
>>>>>>>>>>
>>>>>>>>>> Jakub, can you give a more detailed description what you were
>>>>>>>>>> doing
>>>>>>>>>> with oVirt when this happened? Maybe the bug is still there.
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>> Laszlo
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>>>>>> <nslomian at redhat.com>, users at ovirt.org
>>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM
>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>
>>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI
>>>>>>>>>>> NullPointerException.
>>>>>>>>>>>
>>>>>>>>>>> Greg
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>, "Noam Slomianko"
>>>>>>>>>>>> <nslomian at redhat.com>, "Greg Sheremeta"
>>>>>>>>>>>> <gshereme at redhat.com>
>>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM
>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>
>>>>>>>>>>>> Well if they are no longer in DB then that explains why the
>>>>>>>>>>>> exception no
>>>>>>>>>>>> longer occurs, but at the cost of database corruption.
>>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>> Laszlo
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme at redhat.com>
>>>>>>>>>>>>> Cc: users at ovirt.org, "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
>>>>>>>>>>>>>> It could also be this bug[1], for which I just submitted a
>>>>>>>>>>>>>> fix.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without the
>>>>>>>>>>>>>> quotes) in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> search bar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Greg
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak at redhat.com>
>>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Jakub,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> DB?
>>>>>>>>>>>>>>> select * from vm_static;
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> curl -u admin at internal:blablabla [engine-url]api/vms
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems Noam fixed this issue already in
>>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>> Laszlo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner at nbu.cz>
>>>>>>>>>>>>>>>> To: users at ovirt.org
>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs
>>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too.
>>>>>>>>>>>>>>>>> Repeating
>>>>>>>>>>>>>>>>> problem in log is:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN
>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS ,
>>>>>>>>>>>>>>>>> vds =
>>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error =
>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN
>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS ,
>>>>>>>>>>>>>>>>> vds =
>>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error =
>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>>>> I restarted management node and both nodes and this problem
>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>> But I still can not see any VMs.
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>> Users at ovirt.org
>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not have them in database. Both commands shows nothing
>>>>>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great
>>>>>>>>> (except SPICE. VNC and creating new WM due to serialization bug.).
>>>>>>>>> Because those bugs were irritating, we switched to nightly. Still
>>>>>>>>> works fine (but bug with VMs is not fixed there), than I installed
>>>>>>>>> new
>>>>>>>>> ovirt engine management server in KVM virtualization (using
>>>>>>>>> libvirtd)
>>>>>>>>> due to taking snapshots before deploying nightly builds. New
>>>>>>>>> virtualized engine works great and I delete old one. Next day I
>>>>>>>>> came
>>>>>>>>> to work and engine web page was very very slow, every operation
>>>>>>>>> takes
>>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I
>>>>>>>>> converted
>>>>>>>>> its qcow2 image to raw, because I think it has slow DB because of
>>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes
>>>>>>>>> logs and
>>>>>>>>> found this problem, which I posted in first email (VMs still
>>>>>>>>> running,
>>>>>>>>> but not visible in engine VMs tab, but on host I can see number of
>>>>>>>>> running VMs). After restart bug disappeared, but VMs too.
>>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are
>>>>>>>>> gone. Even from storage. (but disappearing from storage may be my
>>>>>>>>> fault during recovery).
>>>>>>>>>
>>>>>>>>> I can provide logs from both two nodes and (hopefully) from engine
>>>>>>>>> too. I can provide you with those logs, but not publicly. Sorry.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>> Jakub Bittner, NSA Czech
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>
>>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone
>>>>>>>> again.
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>> can you verify the db from before the upgrade has the VMs?
>>>>>> I dont know what you mean by verify database. Please, could you
>>>>>> point me
>>>>>> how to do it? Thank you
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>> I assume from your expectation to see the VMs from the previous
>>>>> install that you upgraded your system rather than do a clean install?
>>>>> in an upgrade, there should be a backup of the db from before the
>>>>> upgrade.
>>>>> (in a clean install, engine is not aware of VMs running on hosts from
>>>>> previous installs - we need to cover "import existing storage domain"
>>>>> in a future version to close this gap. current alternatives around
>>>>> this are to convert the storage domain to an export domain (for an nfs
>>>>> data domain) an re-import them to the system or a few other options.
>>>>>
>>>>> how many VMs are there?
>>>> They(VMs) are deleted. So, my situation is this:
>>>>
>>>> I have clean install of ovirt 3.3.0beta with imported VMs from export
>>>> domain. Everything works except snapshots. But when I restart ovirt mgmt
>>>> server, VMs will disappear. Right now I am doing hourly backup of DB.
>>> this is very fishy...
>>> can you arrange for a remote access session for someone to take a look?
>>>
>>>> I can not use workaround with converting domain to export domain,
>>>> because we use iscsi. It would be perfect if I can retake existing
>>>> domain. Another gap is if I dont detach export domain I can not attach
>>>> it to new datacenter and I have to manually edit some files. It would be
>>>> absolutely perfect to do it from ovirt web gui (for example by "force
>>>> override button")
>>> i thought 3.3 allowed for that, but i could be wrong.
>>
>> Well, I will backup all VMs to export domain, than create snapshot of
>> ovirt and I will try to restart that server if its going to delete VMs
>> from DB. Than I will report back. I think, that remote session will not
>> be possible, but I can be wrong (I hope).
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
I change it. But I can not restart postgres.
More information about the Users
mailing list