Re: [Users] All VMs disappeared

14 Aug 2013


      Dne 14.8.2013 09:57, Liron Aravot napsal(a):
...
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
...
From: "Laszlo Hornyak" <lhornyak@redhat.com>
To: "Jakub Bittner" <j.bittner@nbu.cz>
Cc: users@ovirt.org
Sent: Wednesday, August 14, 2013 10:39:48 AM
Subject: Re: [Users] All VMs disappeared
----- Original Message -----
...
From: "Jakub Bittner" <j.bittner@nbu.cz>
To: users@ovirt.org
Sent: Wednesday, August 14, 2013 8:09:45 AM
Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
...
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql
before you try again?
It is in /var/lib/pgsql/data/postgresql.conf
log_statement = 'all'
----- Original Message -----
...
From: "Jakub Bittner" <j.bittner@nbu.cz>
To: "Itamar Heim" <iheim@redhat.com>
Cc: users@ovirt.org
Sent: Tuesday, August 13, 2013 5:20:29 PM
Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
...
On 08/13/2013 05:31 PM, Jakub Bittner wrote:
> Dne 13.8.2013 15:39, Itamar Heim napsal(a):
>> On 08/13/2013 08:50 AM, Jakub Bittner wrote:
>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a):
>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote:
>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
>>>>>>> I looked around Noam's patch and that should not cause such
>>>>>>> behavior.
>>>>>>> I am wondering how that lost VM's could happen.
>>>>>>>
>>>>>>> Jakub, can you give a more detailed description what you were
>>>>>>> doing
>>>>>>> with oVirt when this happened? Maybe the bug is still there.
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Laszlo
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com>
>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com>
>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko"
>>>>>>>> <nslomian@redhat.com>, users@ovirt.org
>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM
>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>
>>>>>>>> Not the one I fixed, 987907. It was a simple UI
>>>>>>>> NullPointerException.
>>>>>>>>
>>>>>>>> Greg
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com>
>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko"
>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta"
>>>>>>>>> <gshereme@redhat.com>
>>>>>>>>> Cc: users@ovirt.org
>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM
>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>
>>>>>>>>> Well if they are no longer in DB then that explains why the
>>>>>>>>> exception no
>>>>>>>>> longer occurs, but at the cost of database corruption.
>>>>>>>>> Noam and Greg, can these bugs cause data corruption?
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>> Laszlo
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz>
>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com>
>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com>
>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM
>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>
>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
>>>>>>>>>>> It could also be this bug[1], for which I just submitted a
>>>>>>>>>>> fix.
>>>>>>>>>>>
>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
>>>>>>>>>>>
>>>>>>>>>>> You can work around it by typing just "VMs:" (without the
>>>>>>>>>>> quotes) in
>>>>>>>>>>> the
>>>>>>>>>>> search bar.
>>>>>>>>>>>
>>>>>>>>>>> Greg
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com>
>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>
>>>>>>>>>>>> Cc: users@ovirt.org
>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM
>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Jakub,
>>>>>>>>>>>>
>>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in
>>>>>>>>>>>> your
>>>>>>>>>>>> DB?
>>>>>>>>>>>> select * from vm_static;
>>>>>>>>>>>> or
>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms
>>>>>>>>>>>>
>>>>>>>>>>>> It seems Noam fixed this issue already in
>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>> Laszlo
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz>
>>>>>>>>>>>>> To: users@ovirt.org
>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM
>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our
>>>>>>>>>>>>>> VMs
>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too.
>>>>>>>>>>>>>> Repeating
>>>>>>>>>>>>>> problem in log is:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS
>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>> vds =
>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error
>>>>>>>>>>>>>> =
>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN
>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS
>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>> vds =
>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error
>>>>>>>>>>>>>> =
>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.:
>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [vdsbroker.jar:]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown
>>>>>>>>>>>>>> Source) [:1.7.0_25]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>>>>>>>> [rt.jar:1.7.0_25]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [scheduler.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>>             at
>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [quartz.jar:]
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>> Users@ovirt.org
>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>> I restarted management node and both nodes and this
>>>>>>>>>>>>> problem
>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>> But I still can not see any VMs.
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> Users@ovirt.org
>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>> Users@ovirt.org
>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I do not have them in database. Both commands shows nothing
>>>>>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was
>>>>>> great
>>>>>> (except SPICE. VNC and creating new WM due to serialization
>>>>>> bug.).
>>>>>> Because those bugs were irritating, we switched to nightly. Still
>>>>>> works fine (but bug with VMs is not fixed there), than I
>>>>>> installed
>>>>>> new
>>>>>> ovirt engine management server in KVM virtualization (using
>>>>>> libvirtd)
>>>>>> due to taking snapshots before deploying nightly builds. New
>>>>>> virtualized engine works great and I delete old one. Next day I
>>>>>> came
>>>>>> to work and engine web page was very very slow, every operation
>>>>>> takes
>>>>>> ages, that time I found there were no VMs, no HDDs and so. I
>>>>>> converted
>>>>>> its qcow2 image to raw, because I think it has slow DB because of
>>>>>> running i qcow2. Did not helped, than I take a look in nodes
>>>>>> logs and
>>>>>> found this problem, which I posted in first email (VMs still
>>>>>> running,
>>>>>> but not visible in engine VMs tab, but on host I can see number
>>>>>> of
>>>>>> running VMs). After restart bug disappeared, but VMs too.
>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs
>>>>>> are
>>>>>> gone. Even from storage. (but disappearing from storage may be my
>>>>>> fault during recovery).
>>>>>>
>>>>>> I can provide logs from both two nodes and (hopefully) from
>>>>>> engine
>>>>>> too. I can provide you with those logs, but not publicly. Sorry.
>>>>>>
>>>>>> Thank you,
>>>>>> Jakub Bittner, NSA Czech
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users@ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are
>>>>> gone
>>>>> again.
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users@ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>> can you verify the db from before the upgrade has the VMs?
>>> I dont know what you mean by verify database. Please, could you
>>> point me
>>> how to do it? Thank you
>>> _______________________________________________
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>> I assume from your expectation to see the VMs from the previous
>> install that you upgraded your system rather than do a clean install?
>> in an upgrade, there should be a backup of the db from before the
>> upgrade.
>> (in a clean install, engine is not aware of VMs running on hosts from
>> previous installs - we need to cover "import existing storage domain"
>> in a future version to close this gap. current alternatives around
>> this are to convert the storage domain to an export domain (for an
>> nfs
>> data domain) an re-import them to the system or a few other options.
>>
>> how many VMs are there?
> They(VMs) are deleted. So, my situation is this:
>
> I have clean install of ovirt 3.3.0beta with imported VMs from export
> domain. Everything works except snapshots. But when I restart ovirt
> mgmt
> server, VMs will disappear. Right now I am doing hourly backup of DB.
this is very fishy...
can you arrange for a remote access session for someone to take a look?
> I can not use workaround with converting domain to export domain,
> because we use iscsi. It would be perfect if I can retake existing
> domain. Another gap is if I dont detach export domain I can not attach
> it to new datacenter and I have to manually edit some files. It would
> be
> absolutely perfect to do it from ovirt web gui (for example by "force
> override button")
i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of
ovirt and I will try to restart that server if its going to delete VMs
from DB. Than I will report back. I think, that remote session will not
be possible, but I can be wrong (I hope).
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres.
You can restart it only when the engine is down, it will not restart with
database connections open.
...
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt 
deleted them as I thought that it can import them back.