[Users] All VMs disappeared
Hello, Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is: 2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] 2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
Hello,
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs.
Hi Jakub, Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829. Thank you, Laszlo ----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Friday, August 9, 2013 1:06:53 PM Subject: Re: [Users] All VMs disappeared
Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
Hello,
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
It could also be this bug[1], for which I just submitted a fix. [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 You can work around it by typing just "VMs:" (without the quotes) in the search bar. Greg ----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Friday, August 9, 2013 11:21:26 AM Subject: Re: [Users] All VMs disappeared
Hi Jakub,
Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms
It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Friday, August 9, 2013 1:06:53 PM Subject: Re: [Users] All VMs disappeared
Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
Hello,
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
It could also be this bug[1], for which I just submitted a fix.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
You can work around it by typing just "VMs:" (without the quotes) in the search bar.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Friday, August 9, 2013 11:21:26 AM Subject: Re: [Users] All VMs disappeared
Hi Jakub,
Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms
It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Friday, August 9, 2013 1:06:53 PM Subject: Re: [Users] All VMs disappeared
Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
Hello,
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi,
I do not have them in database. Both commands shows nothing
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption? Thank you, Laszlo ----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Monday, August 12, 2013 9:05:09 AM Subject: Re: [Users] All VMs disappeared
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
It could also be this bug[1], for which I just submitted a fix.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
You can work around it by typing just "VMs:" (without the quotes) in the search bar.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Friday, August 9, 2013 11:21:26 AM Subject: Re: [Users] All VMs disappeared
Hi Jakub,
Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms
It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Friday, August 9, 2013 1:06:53 PM Subject: Re: [Users] All VMs disappeared
Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
Hello,
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi,
I do not have them in database. Both commands shows nothing
Not the one I fixed, 987907. It was a simple UI NullPointerException. Greg ----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org Sent: Monday, August 12, 2013 7:21:26 AM Subject: Re: [Users] All VMs disappeared
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption?
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Monday, August 12, 2013 9:05:09 AM Subject: Re: [Users] All VMs disappeared
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
It could also be this bug[1], for which I just submitted a fix.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
You can work around it by typing just "VMs:" (without the quotes) in the search bar.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Friday, August 9, 2013 11:21:26 AM Subject: Re: [Users] All VMs disappeared
Hi Jakub,
Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms
It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Friday, August 9, 2013 1:06:53 PM Subject: Re: [Users] All VMs disappeared
Dne 9.8.2013 09:48, Jakub Bittner napsal(a):
Hello,
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
2013-08-09 09:44:24,203 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-08-09 09:44:27,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = java.lang.NullPointerException, continuing.: java.lang.NullPointerException at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) [vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi,
I do not have them in database. Both commands shows nothing
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen. Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there. Thank you, Laszlo ----- Original Message -----
From: "Greg Sheremeta" <gshereme@redhat.com> To: "Laszlo Hornyak" <lhornyak@redhat.com> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, users@ovirt.org Sent: Monday, August 12, 2013 1:23:40 PM Subject: Re: [Users] All VMs disappeared
Not the one I fixed, 987907. It was a simple UI NullPointerException.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org Sent: Monday, August 12, 2013 7:21:26 AM Subject: Re: [Users] All VMs disappeared
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption?
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Monday, August 12, 2013 9:05:09 AM Subject: Re: [Users] All VMs disappeared
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
It could also be this bug[1], for which I just submitted a fix.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
You can work around it by typing just "VMs:" (without the quotes) in the search bar.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Friday, August 9, 2013 11:21:26 AM Subject: Re: [Users] All VMs disappeared
Hi Jakub,
Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms
It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Friday, August 9, 2013 1:06:53 PM Subject: Re: [Users] All VMs disappeared
Dne 9.8.2013 09:48, Jakub Bittner napsal(a): > Hello, > > Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs > disappeared from VMs tab. Nodes running on centos too. Repeating > problem in log is: > > > 2013-08-09 09:44:24,203 WARN > [org.ovirt.engine.core.vdsbroker.VdsManager] > (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = > 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = > java.lang.NullPointerException, continuing.: > java.lang.NullPointerException > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) > [vdsbroker.jar:] > > at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown > Source) [:1.7.0_25] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.7.0_25] > at java.lang.reflect.Method.invoke(Method.java:606) > [rt.jar:1.7.0_25] > at > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > [quartz.jar:] > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > [quartz.jar:] > > 2013-08-09 09:44:27,039 WARN > [org.ovirt.engine.core.vdsbroker.VdsManager] > (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = > 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = > java.lang.NullPointerException, continuing.: > java.lang.NullPointerException > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) > [vdsbroker.jar:] > > at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown > Source) [:1.7.0_25] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.7.0_25] > at java.lang.reflect.Method.invoke(Method.java:606) > [rt.jar:1.7.0_25] > at > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > [quartz.jar:] > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > [quartz.jar:] > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users I restarted management node and both nodes and this problem disappeared. But I still can not see any VMs. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi,
I do not have them in database. Both commands shows nothing
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen.
Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there.
Thank you, Laszlo
----- Original Message -----
From: "Greg Sheremeta" <gshereme@redhat.com> To: "Laszlo Hornyak" <lhornyak@redhat.com> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, users@ovirt.org Sent: Monday, August 12, 2013 1:23:40 PM Subject: Re: [Users] All VMs disappeared
Not the one I fixed, 987907. It was a simple UI NullPointerException.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org Sent: Monday, August 12, 2013 7:21:26 AM Subject: Re: [Users] All VMs disappeared
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption?
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Monday, August 12, 2013 9:05:09 AM Subject: Re: [Users] All VMs disappeared
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
It could also be this bug[1], for which I just submitted a fix.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
You can work around it by typing just "VMs:" (without the quotes) in the search bar.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Friday, August 9, 2013 11:21:26 AM Subject: Re: [Users] All VMs disappeared
Hi Jakub,
Could you check through DB or REST-API if the VM's are in your DB? select * from vm_static; or curl -u admin@internal:blablabla [engine-url]api/vms
It seems Noam fixed this issue already in c2295c31fa645e1ba1b94cd557bd1fecb40c8829.
Thank you, Laszlo
----- Original Message ----- > From: "Jakub Bittner" <j.bittner@nbu.cz> > To: users@ovirt.org > Sent: Friday, August 9, 2013 1:06:53 PM > Subject: Re: [Users] All VMs disappeared > > Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >> Hello, >> >> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >> disappeared from VMs tab. Nodes running on centos too. Repeating >> problem in log is: >> >> >> 2013-08-09 09:44:24,203 WARN >> [org.ovirt.engine.core.vdsbroker.VdsManager] >> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = >> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >> java.lang.NullPointerException, continuing.: >> java.lang.NullPointerException >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >> [vdsbroker.jar:] >> >> at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >> Source) [:1.7.0_25] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> [rt.jar:1.7.0_25] >> at java.lang.reflect.Method.invoke(Method.java:606) >> [rt.jar:1.7.0_25] >> at >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >> [scheduler.jar:] >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> [quartz.jar:] >> at >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> [quartz.jar:] >> >> 2013-08-09 09:44:27,039 WARN >> [org.ovirt.engine.core.vdsbroker.VdsManager] >> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = >> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >> java.lang.NullPointerException, continuing.: >> java.lang.NullPointerException >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >> [vdsbroker.jar:] >> >> at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >> Source) [:1.7.0_25] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> [rt.jar:1.7.0_25] >> at java.lang.reflect.Method.invoke(Method.java:606) >> [rt.jar:1.7.0_25] >> at >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >> [scheduler.jar:] >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> [quartz.jar:] >> at >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> [quartz.jar:] >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > I restarted management node and both nodes and this problem > disappeared. > But I still can not see any VMs. > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi,
I do not have them in database. Both commands shows nothing
Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery). I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry. Thank you, Jakub Bittner, NSA Czech
Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen.
Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there.
Thank you, Laszlo
----- Original Message -----
From: "Greg Sheremeta" <gshereme@redhat.com> To: "Laszlo Hornyak" <lhornyak@redhat.com> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, users@ovirt.org Sent: Monday, August 12, 2013 1:23:40 PM Subject: Re: [Users] All VMs disappeared
Not the one I fixed, 987907. It was a simple UI NullPointerException.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org Sent: Monday, August 12, 2013 7:21:26 AM Subject: Re: [Users] All VMs disappeared
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption?
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Monday, August 12, 2013 9:05:09 AM Subject: Re: [Users] All VMs disappeared
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a):
It could also be this bug[1], for which I just submitted a fix.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=987907
You can work around it by typing just "VMs:" (without the quotes) in the search bar.
Greg
----- Original Message ----- > From: "Laszlo Hornyak" <lhornyak@redhat.com> > To: "Jakub Bittner" <j.bittner@nbu.cz> > Cc: users@ovirt.org > Sent: Friday, August 9, 2013 11:21:26 AM > Subject: Re: [Users] All VMs disappeared > > Hi Jakub, > > Could you check through DB or REST-API if the VM's are in your DB? > select * from vm_static; > or > curl -u admin@internal:blablabla [engine-url]api/vms > > It seems Noam fixed this issue already in > c2295c31fa645e1ba1b94cd557bd1fecb40c8829. > > Thank you, > Laszlo > > ----- Original Message ----- >> From: "Jakub Bittner" <j.bittner@nbu.cz> >> To: users@ovirt.org >> Sent: Friday, August 9, 2013 1:06:53 PM >> Subject: Re: [Users] All VMs disappeared >> >> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>> Hello, >>> >>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>> disappeared from VMs tab. Nodes running on centos too. Repeating >>> problem in log is: >>> >>> >>> 2013-08-09 09:44:24,203 WARN >>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = >>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>> java.lang.NullPointerException, continuing.: >>> java.lang.NullPointerException >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>> >>> [vdsbroker.jar:] >>> >>> at >>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>> Source) [:1.7.0_25] >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> [rt.jar:1.7.0_25] >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> [rt.jar:1.7.0_25] >>> at >>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>> >>> [scheduler.jar:] >>> at >>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>> [quartz.jar:] >>> at >>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>> >>> [quartz.jar:] >>> >>> 2013-08-09 09:44:27,039 WARN >>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = >>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>> java.lang.NullPointerException, continuing.: >>> java.lang.NullPointerException >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>> >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>> >>> [vdsbroker.jar:] >>> >>> at >>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>> Source) [:1.7.0_25] >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> [rt.jar:1.7.0_25] >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> [rt.jar:1.7.0_25] >>> at >>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>> >>> [scheduler.jar:] >>> at >>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>> [quartz.jar:] >>> at >>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>> >>> [quartz.jar:] >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> I restarted management node and both nodes and this problem >> disappeared. >> But I still can not see any VMs. >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > Hi,
I do not have them in database. Both commands shows nothing
Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery).
I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry.
Thank you, Jakub Bittner, NSA Czech _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again.
On 08/12/2013 06:59 PM, Jakub Bittner wrote:
Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen.
Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there.
Thank you, Laszlo
----- Original Message -----
From: "Greg Sheremeta" <gshereme@redhat.com> To: "Laszlo Hornyak" <lhornyak@redhat.com> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, users@ovirt.org Sent: Monday, August 12, 2013 1:23:40 PM Subject: Re: [Users] All VMs disappeared
Not the one I fixed, 987907. It was a simple UI NullPointerException.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org Sent: Monday, August 12, 2013 7:21:26 AM Subject: Re: [Users] All VMs disappeared
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption?
Thank you, Laszlo
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Monday, August 12, 2013 9:05:09 AM Subject: Re: [Users] All VMs disappeared
Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): > It could also be this bug[1], for which I just submitted a fix. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 > > You can work around it by typing just "VMs:" (without the quotes) in > the > search bar. > > Greg > > > ----- Original Message ----- >> From: "Laszlo Hornyak" <lhornyak@redhat.com> >> To: "Jakub Bittner" <j.bittner@nbu.cz> >> Cc: users@ovirt.org >> Sent: Friday, August 9, 2013 11:21:26 AM >> Subject: Re: [Users] All VMs disappeared >> >> Hi Jakub, >> >> Could you check through DB or REST-API if the VM's are in your DB? >> select * from vm_static; >> or >> curl -u admin@internal:blablabla [engine-url]api/vms >> >> It seems Noam fixed this issue already in >> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >> >> Thank you, >> Laszlo >> >> ----- Original Message ----- >>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>> To: users@ovirt.org >>> Sent: Friday, August 9, 2013 1:06:53 PM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>> Hello, >>>> >>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>> disappeared from VMs tab. Nodes running on centos too. Repeating >>>> problem in log is: >>>> >>>> >>>> 2013-08-09 09:44:24,203 WARN >>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , vds = >>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>> java.lang.NullPointerException, continuing.: >>>> java.lang.NullPointerException >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>> >>>> [vdsbroker.jar:] >>>> >>>> at >>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>> Source) [:1.7.0_25] >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>> [rt.jar:1.7.0_25] >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> [rt.jar:1.7.0_25] >>>> at >>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>> >>>> [scheduler.jar:] >>>> at >>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>> [quartz.jar:] >>>> at >>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>> >>>> [quartz.jar:] >>>> >>>> 2013-08-09 09:44:27,039 WARN >>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , vds = >>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>> java.lang.NullPointerException, continuing.: >>>> java.lang.NullPointerException >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>> >>>> [vdsbroker.jar:] >>>> at >>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>> >>>> [vdsbroker.jar:] >>>> >>>> at >>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>> Source) [:1.7.0_25] >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>> [rt.jar:1.7.0_25] >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> [rt.jar:1.7.0_25] >>>> at >>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>> >>>> [scheduler.jar:] >>>> at >>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>> [quartz.jar:] >>>> at >>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>> >>>> [quartz.jar:] >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> I restarted management node and both nodes and this problem >>> disappeared. >>> But I still can not see any VMs. >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> Hi,
I do not have them in database. Both commands shows nothing
Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery).
I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry.
Thank you, Jakub Bittner, NSA Czech _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote:
Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen.
Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there.
Thank you, Laszlo
----- Original Message -----
From: "Greg Sheremeta" <gshereme@redhat.com> To: "Laszlo Hornyak" <lhornyak@redhat.com> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, users@ovirt.org Sent: Monday, August 12, 2013 1:23:40 PM Subject: Re: [Users] All VMs disappeared
Not the one I fixed, 987907. It was a simple UI NullPointerException.
Greg
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, "Greg Sheremeta" <gshereme@redhat.com> Cc: users@ovirt.org Sent: Monday, August 12, 2013 7:21:26 AM Subject: Re: [Users] All VMs disappeared
Well if they are no longer in DB then that explains why the exception no longer occurs, but at the cost of database corruption. Noam and Greg, can these bugs cause data corruption?
Thank you, Laszlo
----- Original Message ----- > From: "Jakub Bittner" <j.bittner@nbu.cz> > To: "Greg Sheremeta" <gshereme@redhat.com> > Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> > Sent: Monday, August 12, 2013 9:05:09 AM > Subject: Re: [Users] All VMs disappeared > > Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >> It could also be this bug[1], for which I just submitted a fix. >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >> >> You can work around it by typing just "VMs:" (without the >> quotes) in >> the >> search bar. >> >> Greg >> >> >> ----- Original Message ----- >>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>> Cc: users@ovirt.org >>> Sent: Friday, August 9, 2013 11:21:26 AM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Hi Jakub, >>> >>> Could you check through DB or REST-API if the VM's are in your >>> DB? >>> select * from vm_static; >>> or >>> curl -u admin@internal:blablabla [engine-url]api/vms >>> >>> It seems Noam fixed this issue already in >>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>> >>> Thank you, >>> Laszlo >>> >>> ----- Original Message ----- >>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>> To: users@ovirt.org >>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>> Hello, >>>>> >>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>> disappeared from VMs tab. Nodes running on centos too. >>>>> Repeating >>>>> problem in log is: >>>>> >>>>> >>>>> 2013-08-09 09:44:24,203 WARN >>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>> vds = >>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>> java.lang.NullPointerException, continuing.: >>>>> java.lang.NullPointerException >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> >>>>> at >>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>> Source) [:1.7.0_25] >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> >>>>> >>>>> [rt.jar:1.7.0_25] >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> [rt.jar:1.7.0_25] >>>>> at >>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>> >>>>> >>>>> [scheduler.jar:] >>>>> at >>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>> [quartz.jar:] >>>>> at >>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>> >>>>> >>>>> [quartz.jar:] >>>>> >>>>> 2013-08-09 09:44:27,039 WARN >>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>> vds = >>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>> java.lang.NullPointerException, continuing.: >>>>> java.lang.NullPointerException >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> at >>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>> >>>>> >>>>> [vdsbroker.jar:] >>>>> >>>>> at >>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>> Source) [:1.7.0_25] >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> >>>>> >>>>> [rt.jar:1.7.0_25] >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> [rt.jar:1.7.0_25] >>>>> at >>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>> >>>>> >>>>> [scheduler.jar:] >>>>> at >>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>> [quartz.jar:] >>>>> at >>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>> >>>>> >>>>> [quartz.jar:] >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> I restarted management node and both nodes and this problem >>>> disappeared. >>>> But I still can not see any VMs. >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> > Hi, > > I do not have them in database. Both commands shows nothing > Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery).
I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry.
Thank you, Jakub Bittner, NSA Czech _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you
On 08/13/2013 08:50 AM, Jakub Bittner wrote:
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote:
Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen.
Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there.
Thank you, Laszlo
----- Original Message -----
From: "Greg Sheremeta" <gshereme@redhat.com> To: "Laszlo Hornyak" <lhornyak@redhat.com> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" <nslomian@redhat.com>, users@ovirt.org Sent: Monday, August 12, 2013 1:23:40 PM Subject: Re: [Users] All VMs disappeared
Not the one I fixed, 987907. It was a simple UI NullPointerException.
Greg
----- Original Message ----- > From: "Laszlo Hornyak" <lhornyak@redhat.com> > To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" > <nslomian@redhat.com>, "Greg Sheremeta" > <gshereme@redhat.com> > Cc: users@ovirt.org > Sent: Monday, August 12, 2013 7:21:26 AM > Subject: Re: [Users] All VMs disappeared > > Well if they are no longer in DB then that explains why the > exception no > longer occurs, but at the cost of database corruption. > Noam and Greg, can these bugs cause data corruption? > > Thank you, > Laszlo > > ----- Original Message ----- >> From: "Jakub Bittner" <j.bittner@nbu.cz> >> To: "Greg Sheremeta" <gshereme@redhat.com> >> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >> Sent: Monday, August 12, 2013 9:05:09 AM >> Subject: Re: [Users] All VMs disappeared >> >> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>> It could also be this bug[1], for which I just submitted a fix. >>> >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>> >>> You can work around it by typing just "VMs:" (without the >>> quotes) in >>> the >>> search bar. >>> >>> Greg >>> >>> >>> ----- Original Message ----- >>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>> Cc: users@ovirt.org >>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Hi Jakub, >>>> >>>> Could you check through DB or REST-API if the VM's are in your >>>> DB? >>>> select * from vm_static; >>>> or >>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>> >>>> It seems Noam fixed this issue already in >>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>> >>>> Thank you, >>>> Laszlo >>>> >>>> ----- Original Message ----- >>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>> To: users@ovirt.org >>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>> Hello, >>>>>> >>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>> Repeating >>>>>> problem in log is: >>>>>> >>>>>> >>>>>> 2013-08-09 09:44:24,203 WARN >>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>> vds = >>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>> java.lang.NullPointerException, continuing.: >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> >>>>>> at >>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>> Source) [:1.7.0_25] >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> >>>>>> >>>>>> [rt.jar:1.7.0_25] >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> [rt.jar:1.7.0_25] >>>>>> at >>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>> >>>>>> >>>>>> [scheduler.jar:] >>>>>> at >>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>> [quartz.jar:] >>>>>> at >>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>> >>>>>> >>>>>> [quartz.jar:] >>>>>> >>>>>> 2013-08-09 09:44:27,039 WARN >>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>> vds = >>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>> java.lang.NullPointerException, continuing.: >>>>>> java.lang.NullPointerException >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> at >>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>> >>>>>> >>>>>> [vdsbroker.jar:] >>>>>> >>>>>> at >>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>> Source) [:1.7.0_25] >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> >>>>>> >>>>>> [rt.jar:1.7.0_25] >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> [rt.jar:1.7.0_25] >>>>>> at >>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>> >>>>>> >>>>>> [scheduler.jar:] >>>>>> at >>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>> [quartz.jar:] >>>>>> at >>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>> >>>>>> >>>>>> [quartz.jar:] >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> I restarted management node and both nodes and this problem >>>>> disappeared. >>>>> But I still can not see any VMs. >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >> Hi, >> >> I do not have them in database. Both commands shows nothing >> Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery).
I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry.
Thank you, Jakub Bittner, NSA Czech _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options. how many VMs are there?
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
On 08/13/2013 08:50 AM, Jakub Bittner wrote:
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote:
Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a):
I looked around Noam's patch and that should not cause such behavior. I am wondering how that lost VM's could happen.
Jakub, can you give a more detailed description what you were doing with oVirt when this happened? Maybe the bug is still there.
Thank you, Laszlo
----- Original Message ----- > From: "Greg Sheremeta" <gshereme@redhat.com> > To: "Laszlo Hornyak" <lhornyak@redhat.com> > Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" > <nslomian@redhat.com>, users@ovirt.org > Sent: Monday, August 12, 2013 1:23:40 PM > Subject: Re: [Users] All VMs disappeared > > Not the one I fixed, 987907. It was a simple UI > NullPointerException. > > Greg > > > ----- Original Message ----- >> From: "Laszlo Hornyak" <lhornyak@redhat.com> >> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >> <nslomian@redhat.com>, "Greg Sheremeta" >> <gshereme@redhat.com> >> Cc: users@ovirt.org >> Sent: Monday, August 12, 2013 7:21:26 AM >> Subject: Re: [Users] All VMs disappeared >> >> Well if they are no longer in DB then that explains why the >> exception no >> longer occurs, but at the cost of database corruption. >> Noam and Greg, can these bugs cause data corruption? >> >> Thank you, >> Laszlo >> >> ----- Original Message ----- >>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>> To: "Greg Sheremeta" <gshereme@redhat.com> >>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>> Sent: Monday, August 12, 2013 9:05:09 AM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>> It could also be this bug[1], for which I just submitted a fix. >>>> >>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>> >>>> You can work around it by typing just "VMs:" (without the >>>> quotes) in >>>> the >>>> search bar. >>>> >>>> Greg >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>> Cc: users@ovirt.org >>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Hi Jakub, >>>>> >>>>> Could you check through DB or REST-API if the VM's are in your >>>>> DB? >>>>> select * from vm_static; >>>>> or >>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>> >>>>> It seems Noam fixed this issue already in >>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>> >>>>> Thank you, >>>>> Laszlo >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>> To: users@ovirt.org >>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>> Hello, >>>>>>> >>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>> Repeating >>>>>>> problem in log is: >>>>>>> >>>>>>> >>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>> vds = >>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>> java.lang.NullPointerException, continuing.: >>>>>>> java.lang.NullPointerException >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> >>>>>>> at >>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>> Source) [:1.7.0_25] >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [rt.jar:1.7.0_25] >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> [rt.jar:1.7.0_25] >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [scheduler.jar:] >>>>>>> at >>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>> [quartz.jar:] >>>>>>> at >>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [quartz.jar:] >>>>>>> >>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>> vds = >>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>> java.lang.NullPointerException, continuing.: >>>>>>> java.lang.NullPointerException >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> at >>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [vdsbroker.jar:] >>>>>>> >>>>>>> at >>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>> Source) [:1.7.0_25] >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [rt.jar:1.7.0_25] >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> [rt.jar:1.7.0_25] >>>>>>> at >>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [scheduler.jar:] >>>>>>> at >>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>> [quartz.jar:] >>>>>>> at >>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>> >>>>>>> >>>>>>> >>>>>>> [quartz.jar:] >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> I restarted management node and both nodes and this problem >>>>>> disappeared. >>>>>> But I still can not see any VMs. >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>> Hi, >>> >>> I do not have them in database. Both commands shows nothing >>> Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery).
I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry.
Thank you, Jakub Bittner, NSA Czech _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB. I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button")
On 08/13/2013 05:31 PM, Jakub Bittner wrote:
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
On 08/13/2013 08:50 AM, Jakub Bittner wrote:
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote:
Dne 12.8.2013 14:54, Jakub Bittner napsal(a):
Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): > I looked around Noam's patch and that should not cause such > behavior. > I am wondering how that lost VM's could happen. > > Jakub, can you give a more detailed description what you were doing > with oVirt when this happened? Maybe the bug is still there. > > Thank you, > Laszlo > > ----- Original Message ----- >> From: "Greg Sheremeta" <gshereme@redhat.com> >> To: "Laszlo Hornyak" <lhornyak@redhat.com> >> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >> <nslomian@redhat.com>, users@ovirt.org >> Sent: Monday, August 12, 2013 1:23:40 PM >> Subject: Re: [Users] All VMs disappeared >> >> Not the one I fixed, 987907. It was a simple UI >> NullPointerException. >> >> Greg >> >> >> ----- Original Message ----- >>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>> <nslomian@redhat.com>, "Greg Sheremeta" >>> <gshereme@redhat.com> >>> Cc: users@ovirt.org >>> Sent: Monday, August 12, 2013 7:21:26 AM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Well if they are no longer in DB then that explains why the >>> exception no >>> longer occurs, but at the cost of database corruption. >>> Noam and Greg, can these bugs cause data corruption? >>> >>> Thank you, >>> Laszlo >>> >>> ----- Original Message ----- >>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>> It could also be this bug[1], for which I just submitted a fix. >>>>> >>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>> >>>>> You can work around it by typing just "VMs:" (without the >>>>> quotes) in >>>>> the >>>>> search bar. >>>>> >>>>> Greg >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>> Cc: users@ovirt.org >>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Hi Jakub, >>>>>> >>>>>> Could you check through DB or REST-API if the VM's are in your >>>>>> DB? >>>>>> select * from vm_static; >>>>>> or >>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>> >>>>>> It seems Noam fixed this issue already in >>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>> >>>>>> Thank you, >>>>>> Laszlo >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>> To: users@ovirt.org >>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>> Hello, >>>>>>>> >>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>> Repeating >>>>>>>> problem in log is: >>>>>>>> >>>>>>>> >>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>> vds = >>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>> java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> >>>>>>>> at >>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>> Source) [:1.7.0_25] >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [rt.jar:1.7.0_25] >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>> [rt.jar:1.7.0_25] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [scheduler.jar:] >>>>>>>> at >>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>> [quartz.jar:] >>>>>>>> at >>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [quartz.jar:] >>>>>>>> >>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>> vds = >>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>> java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [vdsbroker.jar:] >>>>>>>> >>>>>>>> at >>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>> Source) [:1.7.0_25] >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [rt.jar:1.7.0_25] >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>> [rt.jar:1.7.0_25] >>>>>>>> at >>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [scheduler.jar:] >>>>>>>> at >>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>> [quartz.jar:] >>>>>>>> at >>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [quartz.jar:] >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> I restarted management node and both nodes and this problem >>>>>>> disappeared. >>>>>>> But I still can not see any VMs. >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>> Hi, >>>> >>>> I do not have them in database. Both commands shows nothing >>>> Hello,
we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great (except SPICE. VNC and creating new WM due to serialization bug.). Because those bugs were irritating, we switched to nightly. Still works fine (but bug with VMs is not fixed there), than I installed new ovirt engine management server in KVM virtualization (using libvirtd) due to taking snapshots before deploying nightly builds. New virtualized engine works great and I delete old one. Next day I came to work and engine web page was very very slow, every operation takes ages, that time I found there were no VMs, no HDDs and so. I converted its qcow2 image to raw, because I think it has slow DB because of running i qcow2. Did not helped, than I take a look in nodes logs and found this problem, which I posted in first email (VMs still running, but not visible in engine VMs tab, but on host I can see number of running VMs). After restart bug disappeared, but VMs too. Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are gone. Even from storage. (but disappearing from storage may be my fault during recovery).
I can provide logs from both two nodes and (hopefully) from engine too. I can provide you with those logs, but not publicly. Sorry.
Thank you, Jakub Bittner, NSA Czech _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button")
i thought 3.3 allowed for that, but i could be wrong.
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
On 08/13/2013 05:31 PM, Jakub Bittner wrote:
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
On 08/13/2013 08:50 AM, Jakub Bittner wrote:
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote:
Dne 12.8.2013 14:54, Jakub Bittner napsal(a): > Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >> I looked around Noam's patch and that should not cause such >> behavior. >> I am wondering how that lost VM's could happen. >> >> Jakub, can you give a more detailed description what you were >> doing >> with oVirt when this happened? Maybe the bug is still there. >> >> Thank you, >> Laszlo >> >> ----- Original Message ----- >>> From: "Greg Sheremeta" <gshereme@redhat.com> >>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>> <nslomian@redhat.com>, users@ovirt.org >>> Sent: Monday, August 12, 2013 1:23:40 PM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Not the one I fixed, 987907. It was a simple UI >>> NullPointerException. >>> >>> Greg >>> >>> >>> ----- Original Message ----- >>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>> <gshereme@redhat.com> >>>> Cc: users@ovirt.org >>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Well if they are no longer in DB then that explains why the >>>> exception no >>>> longer occurs, but at the cost of database corruption. >>>> Noam and Greg, can these bugs cause data corruption? >>>> >>>> Thank you, >>>> Laszlo >>>> >>>> ----- Original Message ----- >>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>> It could also be this bug[1], for which I just submitted a >>>>>> fix. >>>>>> >>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>> >>>>>> You can work around it by typing just "VMs:" (without the >>>>>> quotes) in >>>>>> the >>>>>> search bar. >>>>>> >>>>>> Greg >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>> Cc: users@ovirt.org >>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Hi Jakub, >>>>>>> >>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>> your >>>>>>> DB? >>>>>>> select * from vm_static; >>>>>>> or >>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>> >>>>>>> It seems Noam fixed this issue already in >>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>> >>>>>>> Thank you, >>>>>>> Laszlo >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>> To: users@ovirt.org >>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>> Repeating >>>>>>>>> problem in log is: >>>>>>>>> >>>>>>>>> >>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>> vds = >>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>> java.lang.NullPointerException >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> >>>>>>>>> at >>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>> Source) [:1.7.0_25] >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>> at >>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [scheduler.jar:] >>>>>>>>> at >>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>> [quartz.jar:] >>>>>>>>> at >>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [quartz.jar:] >>>>>>>>> >>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>> vds = >>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>> java.lang.NullPointerException >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [vdsbroker.jar:] >>>>>>>>> >>>>>>>>> at >>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>> Source) [:1.7.0_25] >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>> at >>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>> at >>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [scheduler.jar:] >>>>>>>>> at >>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>> [quartz.jar:] >>>>>>>>> at >>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [quartz.jar:] >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> I restarted management node and both nodes and this problem >>>>>>>> disappeared. >>>>>>>> But I still can not see any VMs. >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>>> Hi, >>>>> >>>>> I do not have them in database. Both commands shows nothing >>>>> > Hello, > > we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great > (except SPICE. VNC and creating new WM due to serialization bug.). > Because those bugs were irritating, we switched to nightly. Still > works fine (but bug with VMs is not fixed there), than I installed > new > ovirt engine management server in KVM virtualization (using > libvirtd) > due to taking snapshots before deploying nightly builds. New > virtualized engine works great and I delete old one. Next day I > came > to work and engine web page was very very slow, every operation > takes > ages, that time I found there were no VMs, no HDDs and so. I > converted > its qcow2 image to raw, because I think it has slow DB because of > running i qcow2. Did not helped, than I take a look in nodes > logs and > found this problem, which I posted in first email (VMs still > running, > but not visible in engine VMs tab, but on host I can see number of > running VMs). After restart bug disappeared, but VMs too. > Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are > gone. Even from storage. (but disappearing from storage may be my > fault during recovery). > > I can provide logs from both two nodes and (hopefully) from engine > too. I can provide you with those logs, but not publicly. Sorry. > > Thank you, > Jakub Bittner, NSA Czech > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone again. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button")
i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
On 08/13/2013 06:20 PM, Jakub Bittner wrote:
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
On 08/13/2013 05:31 PM, Jakub Bittner wrote:
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
On 08/13/2013 08:50 AM, Jakub Bittner wrote:
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote: > Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>> I looked around Noam's patch and that should not cause such >>> behavior. >>> I am wondering how that lost VM's could happen. >>> >>> Jakub, can you give a more detailed description what you were >>> doing >>> with oVirt when this happened? Maybe the bug is still there. >>> >>> Thank you, >>> Laszlo >>> >>> ----- Original Message ----- >>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>> <nslomian@redhat.com>, users@ovirt.org >>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Not the one I fixed, 987907. It was a simple UI >>>> NullPointerException. >>>> >>>> Greg >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>> <gshereme@redhat.com> >>>>> Cc: users@ovirt.org >>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Well if they are no longer in DB then that explains why the >>>>> exception no >>>>> longer occurs, but at the cost of database corruption. >>>>> Noam and Greg, can these bugs cause data corruption? >>>>> >>>>> Thank you, >>>>> Laszlo >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>> fix. >>>>>>> >>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>> >>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>> quotes) in >>>>>>> the >>>>>>> search bar. >>>>>>> >>>>>>> Greg >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>> Cc: users@ovirt.org >>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Hi Jakub, >>>>>>>> >>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>> your >>>>>>>> DB? >>>>>>>> select * from vm_static; >>>>>>>> or >>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>> >>>>>>>> It seems Noam fixed this issue already in >>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Laszlo >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>> To: users@ovirt.org >>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>> Repeating >>>>>>>>>> problem in log is: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>> vds = >>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [scheduler.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>> [quartz.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [quartz.jar:] >>>>>>>>>> >>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>> vds = >>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [scheduler.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>> [quartz.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [quartz.jar:] >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>> disappeared. >>>>>>>>> But I still can not see any VMs. >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>> Hi, >>>>>> >>>>>> I do not have them in database. Both commands shows nothing >>>>>> >> Hello, >> >> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >> (except SPICE. VNC and creating new WM due to serialization bug.). >> Because those bugs were irritating, we switched to nightly. Still >> works fine (but bug with VMs is not fixed there), than I installed >> new >> ovirt engine management server in KVM virtualization (using >> libvirtd) >> due to taking snapshots before deploying nightly builds. New >> virtualized engine works great and I delete old one. Next day I >> came >> to work and engine web page was very very slow, every operation >> takes >> ages, that time I found there were no VMs, no HDDs and so. I >> converted >> its qcow2 image to raw, because I think it has slow DB because of >> running i qcow2. Did not helped, than I take a look in nodes >> logs and >> found this problem, which I posted in first email (VMs still >> running, >> but not visible in engine VMs tab, but on host I can see number of >> running VMs). After restart bug disappeared, but VMs too. >> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >> gone. Even from storage. (but disappearing from storage may be my >> fault during recovery). >> >> I can provide logs from both two nodes and (hopefully) from engine >> too. I can provide you with those logs, but not publicly. Sorry. >> >> Thank you, >> Jakub Bittner, NSA Czech >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone > again. > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button")
i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
remote session could be via a webex-like, giving control to remote person via the webex-like tool - not direct internet connection to the machine
Ahoj Jakub, Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all' ----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
On 08/13/2013 05:31 PM, Jakub Bittner wrote:
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
On 08/13/2013 08:50 AM, Jakub Bittner wrote:
Dne 12.8.2013 22:44, Itamar Heim napsal(a):
On 08/12/2013 06:59 PM, Jakub Bittner wrote: > Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>> I looked around Noam's patch and that should not cause such >>> behavior. >>> I am wondering how that lost VM's could happen. >>> >>> Jakub, can you give a more detailed description what you were >>> doing >>> with oVirt when this happened? Maybe the bug is still there. >>> >>> Thank you, >>> Laszlo >>> >>> ----- Original Message ----- >>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>> <nslomian@redhat.com>, users@ovirt.org >>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Not the one I fixed, 987907. It was a simple UI >>>> NullPointerException. >>>> >>>> Greg >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>> <gshereme@redhat.com> >>>>> Cc: users@ovirt.org >>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Well if they are no longer in DB then that explains why the >>>>> exception no >>>>> longer occurs, but at the cost of database corruption. >>>>> Noam and Greg, can these bugs cause data corruption? >>>>> >>>>> Thank you, >>>>> Laszlo >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>> fix. >>>>>>> >>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>> >>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>> quotes) in >>>>>>> the >>>>>>> search bar. >>>>>>> >>>>>>> Greg >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>> Cc: users@ovirt.org >>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Hi Jakub, >>>>>>>> >>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>> your >>>>>>>> DB? >>>>>>>> select * from vm_static; >>>>>>>> or >>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>> >>>>>>>> It seems Noam fixed this issue already in >>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Laszlo >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>> To: users@ovirt.org >>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>> Repeating >>>>>>>>>> problem in log is: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>> vds = >>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [scheduler.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>> [quartz.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [quartz.jar:] >>>>>>>>>> >>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>> vds = >>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>> java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>> at >>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [scheduler.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>> [quartz.jar:] >>>>>>>>>> at >>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [quartz.jar:] >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>> disappeared. >>>>>>>>> But I still can not see any VMs. >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>> Hi, >>>>>> >>>>>> I do not have them in database. Both commands shows nothing >>>>>> >> Hello, >> >> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >> (except SPICE. VNC and creating new WM due to serialization bug.). >> Because those bugs were irritating, we switched to nightly. Still >> works fine (but bug with VMs is not fixed there), than I installed >> new >> ovirt engine management server in KVM virtualization (using >> libvirtd) >> due to taking snapshots before deploying nightly builds. New >> virtualized engine works great and I delete old one. Next day I >> came >> to work and engine web page was very very slow, every operation >> takes >> ages, that time I found there were no VMs, no HDDs and so. I >> converted >> its qcow2 image to raw, because I think it has slow DB because of >> running i qcow2. Did not helped, than I take a look in nodes >> logs and >> found this problem, which I posted in first email (VMs still >> running, >> but not visible in engine VMs tab, but on host I can see number of >> running VMs). After restart bug disappeared, but VMs too. >> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >> gone. Even from storage. (but disappearing from storage may be my >> fault during recovery). >> >> I can provide logs from both two nodes and (hopefully) from engine >> too. I can provide you with those logs, but not publicly. Sorry. >> >> Thank you, >> Jakub Bittner, NSA Czech >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone > again. > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
can you verify the db from before the upgrade has the VMs?
I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button")
i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
Dne 12.8.2013 22:44, Itamar Heim napsal(a): > On 08/12/2013 06:59 PM, Jakub Bittner wrote: >> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>> I looked around Noam's patch and that should not cause such >>>> behavior. >>>> I am wondering how that lost VM's could happen. >>>> >>>> Jakub, can you give a more detailed description what you were >>>> doing >>>> with oVirt when this happened? Maybe the bug is still there. >>>> >>>> Thank you, >>>> Laszlo >>>> >>>> ----- Original Message ----- >>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>> <nslomian@redhat.com>, users@ovirt.org >>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Not the one I fixed, 987907. It was a simple UI >>>>> NullPointerException. >>>>> >>>>> Greg >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>> <gshereme@redhat.com> >>>>>> Cc: users@ovirt.org >>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Well if they are no longer in DB then that explains why the >>>>>> exception no >>>>>> longer occurs, but at the cost of database corruption. >>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>> >>>>>> Thank you, >>>>>> Laszlo >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>> fix. >>>>>>>> >>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>> >>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>> quotes) in >>>>>>>> the >>>>>>>> search bar. >>>>>>>> >>>>>>>> Greg >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>> Cc: users@ovirt.org >>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Hi Jakub, >>>>>>>>> >>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>> your >>>>>>>>> DB? >>>>>>>>> select * from vm_static; >>>>>>>>> or >>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>> >>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Laszlo >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>> To: users@ovirt.org >>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>> Repeating >>>>>>>>>>> problem in log is: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>>> vds = >>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> >>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>>> vds = >>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>> at >>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> at >>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [quartz.jar:] >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>>> disappeared. >>>>>>>>>> But I still can not see any VMs. >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I do not have them in database. Both commands shows nothing >>>>>>> >>> Hello, >>> >>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >>> (except SPICE. VNC and creating new WM due to serialization bug.). >>> Because those bugs were irritating, we switched to nightly. Still >>> works fine (but bug with VMs is not fixed there), than I installed >>> new >>> ovirt engine management server in KVM virtualization (using >>> libvirtd) >>> due to taking snapshots before deploying nightly builds. New >>> virtualized engine works great and I delete old one. Next day I >>> came >>> to work and engine web page was very very slow, every operation >>> takes >>> ages, that time I found there were no VMs, no HDDs and so. I >>> converted >>> its qcow2 image to raw, because I think it has slow DB because of >>> running i qcow2. Did not helped, than I take a look in nodes >>> logs and >>> found this problem, which I posted in first email (VMs still >>> running, >>> but not visible in engine VMs tab, but on host I can see number of >>> running VMs). After restart bug disappeared, but VMs too. >>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >>> gone. Even from storage. (but disappearing from storage may be my >>> fault during recovery). >>> >>> I can provide logs from both two nodes and (hopefully) from engine >>> too. I can provide you with those logs, but not publicly. Sorry. >>> >>> Thank you, >>> Jakub Bittner, NSA Czech >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone >> again. >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > can you verify the db from before the upgrade has the VMs? I dont know what you mean by verify database. Please, could you point me how to do it? Thank you _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from
On 08/13/2013 08:50 AM, Jakub Bittner wrote: previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
On 08/13/2013 05:31 PM, Jakub Bittner wrote: this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button") i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres.
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
Dne 13.8.2013 15:39, Itamar Heim napsal(a):
On 08/13/2013 08:50 AM, Jakub Bittner wrote: > Dne 12.8.2013 22:44, Itamar Heim napsal(a): >> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>> I looked around Noam's patch and that should not cause such >>>>> behavior. >>>>> I am wondering how that lost VM's could happen. >>>>> >>>>> Jakub, can you give a more detailed description what you were >>>>> doing >>>>> with oVirt when this happened? Maybe the bug is still there. >>>>> >>>>> Thank you, >>>>> Laszlo >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>> Subject: Re: [Users] All VMs disappeared >>>>>> >>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>> NullPointerException. >>>>>> >>>>>> Greg >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>> <gshereme@redhat.com> >>>>>>> Cc: users@ovirt.org >>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Well if they are no longer in DB then that explains why the >>>>>>> exception no >>>>>>> longer occurs, but at the cost of database corruption. >>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>> >>>>>>> Thank you, >>>>>>> Laszlo >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>>> fix. >>>>>>>>> >>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>> >>>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>>> quotes) in >>>>>>>>> the >>>>>>>>> search bar. >>>>>>>>> >>>>>>>>> Greg >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Hi Jakub, >>>>>>>>>> >>>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>>> your >>>>>>>>>> DB? >>>>>>>>>> select * from vm_static; >>>>>>>>>> or >>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>> >>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs >>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>>> Repeating >>>>>>>>>>>> problem in log is: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS , >>>>>>>>>>>> vds = >>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error = >>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> >>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS , >>>>>>>>>>>> vds = >>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error = >>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> at >>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> I restarted management node and both nodes and this problem >>>>>>>>>>> disappeared. >>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I do not have them in database. Both commands shows nothing >>>>>>>> >>>> Hello, >>>> >>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was great >>>> (except SPICE. VNC and creating new WM due to serialization bug.). >>>> Because those bugs were irritating, we switched to nightly. Still >>>> works fine (but bug with VMs is not fixed there), than I installed >>>> new >>>> ovirt engine management server in KVM virtualization (using >>>> libvirtd) >>>> due to taking snapshots before deploying nightly builds. New >>>> virtualized engine works great and I delete old one. Next day I >>>> came >>>> to work and engine web page was very very slow, every operation >>>> takes >>>> ages, that time I found there were no VMs, no HDDs and so. I >>>> converted >>>> its qcow2 image to raw, because I think it has slow DB because of >>>> running i qcow2. Did not helped, than I take a look in nodes >>>> logs and >>>> found this problem, which I posted in first email (VMs still >>>> running, >>>> but not visible in engine VMs tab, but on host I can see number of >>>> running VMs). After restart bug disappeared, but VMs too. >>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs are >>>> gone. Even from storage. (but disappearing from storage may be my >>>> fault during recovery). >>>> >>>> I can provide logs from both two nodes and (hopefully) from engine >>>> too. I can provide you with those logs, but not publicly. Sorry. >>>> >>>> Thank you, >>>> Jakub Bittner, NSA Czech >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are gone >>> again. >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> can you verify the db from before the upgrade has the VMs? > I dont know what you mean by verify database. Please, could you > point me > how to do it? Thank you > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users I assume from your expectation to see the VMs from the previous install that you upgraded your system rather than do a clean install? in an upgrade, there should be a backup of the db from before the upgrade. (in a clean install, engine is not aware of VMs running on hosts from previous installs - we need to cover "import existing storage domain" in a future version to close this gap. current alternatives around this are to convert the storage domain to an export domain (for an nfs data domain) an re-import them to the system or a few other options.
how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
On 08/13/2013 05:31 PM, Jakub Bittner wrote: this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button") i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres.
You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Jakub, are only your vms disappear? what about the disks? ----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
Dne 13.8.2013 15:39, Itamar Heim napsal(a): > On 08/13/2013 08:50 AM, Jakub Bittner wrote: >> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>> I looked around Noam's patch and that should not cause such >>>>>> behavior. >>>>>> I am wondering how that lost VM's could happen. >>>>>> >>>>>> Jakub, can you give a more detailed description what you were >>>>>> doing >>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>> >>>>>> Thank you, >>>>>> Laszlo >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>> >>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>> NullPointerException. >>>>>>> >>>>>>> Greg >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>> <gshereme@redhat.com> >>>>>>>> Cc: users@ovirt.org >>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Well if they are no longer in DB then that explains why the >>>>>>>> exception no >>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Laszlo >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>>>> fix. >>>>>>>>>> >>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>> >>>>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>>>> quotes) in >>>>>>>>>> the >>>>>>>>>> search bar. >>>>>>>>>> >>>>>>>>>> Greg >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Hi Jakub, >>>>>>>>>>> >>>>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>>>> your >>>>>>>>>>> DB? >>>>>>>>>>> select * from vm_static; >>>>>>>>>>> or >>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>> >>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Laszlo >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our >>>>>>>>>>>>> VMs >>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>>>> Repeating >>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS >>>>>>>>>>>>> , >>>>>>>>>>>>> vds = >>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error >>>>>>>>>>>>> = >>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>> >>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS >>>>>>>>>>>>> , >>>>>>>>>>>>> vds = >>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error >>>>>>>>>>>>> = >>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>> >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>> at >>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>> problem >>>>>>>>>>>> disappeared. >>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I do not have them in database. Both commands shows nothing >>>>>>>>> >>>>> Hello, >>>>> >>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>> great >>>>> (except SPICE. VNC and creating new WM due to serialization >>>>> bug.). >>>>> Because those bugs were irritating, we switched to nightly. Still >>>>> works fine (but bug with VMs is not fixed there), than I >>>>> installed >>>>> new >>>>> ovirt engine management server in KVM virtualization (using >>>>> libvirtd) >>>>> due to taking snapshots before deploying nightly builds. New >>>>> virtualized engine works great and I delete old one. Next day I >>>>> came >>>>> to work and engine web page was very very slow, every operation >>>>> takes >>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>> converted >>>>> its qcow2 image to raw, because I think it has slow DB because of >>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>> logs and >>>>> found this problem, which I posted in first email (VMs still >>>>> running, >>>>> but not visible in engine VMs tab, but on host I can see number >>>>> of >>>>> running VMs). After restart bug disappeared, but VMs too. >>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs >>>>> are >>>>> gone. Even from storage. (but disappearing from storage may be my >>>>> fault during recovery). >>>>> >>>>> I can provide logs from both two nodes and (hopefully) from >>>>> engine >>>>> too. I can provide you with those logs, but not publicly. Sorry. >>>>> >>>>> Thank you, >>>>> Jakub Bittner, NSA Czech >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are >>>> gone >>>> again. >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> can you verify the db from before the upgrade has the VMs? >> I dont know what you mean by verify database. Please, could you >> point me >> how to do it? Thank you >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > I assume from your expectation to see the VMs from the previous > install that you upgraded your system rather than do a clean install? > in an upgrade, there should be a backup of the db from before the > upgrade. > (in a clean install, engine is not aware of VMs running on hosts from > previous installs - we need to cover "import existing storage domain" > in a future version to close this gap. current alternatives around > this are to convert the storage domain to an export domain (for an > nfs > data domain) an re-import them to the system or a few other options. > > how many VMs are there? They(VMs) are deleted. So, my situation is this:
I have clean install of ovirt 3.3.0beta with imported VMs from export domain. Everything works except snapshots. But when I restart ovirt mgmt server, VMs will disappear. Right now I am doing hourly backup of DB.
On 08/13/2013 05:31 PM, Jakub Bittner wrote: this is very fishy... can you arrange for a remote access session for someone to take a look?
I can not use workaround with converting domain to export domain, because we use iscsi. It would be perfect if I can retake existing domain. Another gap is if I dont detach export domain I can not attach it to new datacenter and I have to manually edit some files. It would be absolutely perfect to do it from ovirt web gui (for example by "force override button") i thought 3.3 allowed for that, but i could be wrong.
Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres.
You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a):
On 08/13/2013 05:31 PM, Jakub Bittner wrote: > Dne 13.8.2013 15:39, Itamar Heim napsal(a): >> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>> I looked around Noam's patch and that should not cause such >>>>>>> behavior. >>>>>>> I am wondering how that lost VM's could happen. >>>>>>> >>>>>>> Jakub, can you give a more detailed description what you were >>>>>>> doing >>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>> >>>>>>> Thank you, >>>>>>> Laszlo >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>> >>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>> NullPointerException. >>>>>>>> >>>>>>>> Greg >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>> <gshereme@redhat.com> >>>>>>>>> Cc: users@ovirt.org >>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Well if they are no longer in DB then that explains why the >>>>>>>>> exception no >>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Laszlo >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>> It could also be this bug[1], for which I just submitted a >>>>>>>>>>> fix. >>>>>>>>>>> >>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>> >>>>>>>>>>> You can work around it by typing just "VMs:" (without the >>>>>>>>>>> quotes) in >>>>>>>>>>> the >>>>>>>>>>> search bar. >>>>>>>>>>> >>>>>>>>>>> Greg >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>> >>>>>>>>>>>> Could you check through DB or REST-API if the VM's are in >>>>>>>>>>>> your >>>>>>>>>>>> DB? >>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>> or >>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>> >>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Laszlo >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our >>>>>>>>>>>>>> VMs >>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos too. >>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to refresh VDS >>>>>>>>>>>>>> , >>>>>>>>>>>>>> vds = >>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : node2.x.com, error >>>>>>>>>>>>>> = >>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to refresh VDS >>>>>>>>>>>>>> , >>>>>>>>>>>>>> vds = >>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : node3.x.com, error >>>>>>>>>>>>>> = >>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>> >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>> problem >>>>>>>>>>>>> disappeared. >>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I do not have them in database. Both commands shows nothing >>>>>>>>>> >>>>>> Hello, >>>>>> >>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>> great >>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>> bug.). >>>>>> Because those bugs were irritating, we switched to nightly. Still >>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>> installed >>>>>> new >>>>>> ovirt engine management server in KVM virtualization (using >>>>>> libvirtd) >>>>>> due to taking snapshots before deploying nightly builds. New >>>>>> virtualized engine works great and I delete old one. Next day I >>>>>> came >>>>>> to work and engine web page was very very slow, every operation >>>>>> takes >>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>> converted >>>>>> its qcow2 image to raw, because I think it has slow DB because of >>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>> logs and >>>>>> found this problem, which I posted in first email (VMs still >>>>>> running, >>>>>> but not visible in engine VMs tab, but on host I can see number >>>>>> of >>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my VMs >>>>>> are >>>>>> gone. Even from storage. (but disappearing from storage may be my >>>>>> fault during recovery). >>>>>> >>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>> engine >>>>>> too. I can provide you with those logs, but not publicly. Sorry. >>>>>> >>>>>> Thank you, >>>>>> Jakub Bittner, NSA Czech >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs are >>>>> gone >>>>> again. >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> can you verify the db from before the upgrade has the VMs? >>> I dont know what you mean by verify database. Please, could you >>> point me >>> how to do it? Thank you >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> I assume from your expectation to see the VMs from the previous >> install that you upgraded your system rather than do a clean install? >> in an upgrade, there should be a backup of the db from before the >> upgrade. >> (in a clean install, engine is not aware of VMs running on hosts from >> previous installs - we need to cover "import existing storage domain" >> in a future version to close this gap. current alternatives around >> this are to convert the storage domain to an export domain (for an >> nfs >> data domain) an re-import them to the system or a few other options. >> >> how many VMs are there? > They(VMs) are deleted. So, my situation is this: > > I have clean install of ovirt 3.3.0beta with imported VMs from export > domain. Everything works except snapshots. But when I restart ovirt > mgmt > server, VMs will disappear. Right now I am doing hourly backup of DB. this is very fishy... can you arrange for a remote access session for someone to take a look?
> I can not use workaround with converting domain to export domain, > because we use iscsi. It would be perfect if I can retake existing > domain. Another gap is if I dont detach export domain I can not attach > it to new datacenter and I have to manually edit some files. It would > be > absolutely perfect to do it from ovirt web gui (for example by "force > override button") i thought 3.3 allowed for that, but i could be wrong. Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: "Itamar Heim" <iheim@redhat.com> Cc: users@ovirt.org Sent: Tuesday, August 13, 2013 5:20:29 PM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 17:02, Itamar Heim napsal(a): > On 08/13/2013 05:31 PM, Jakub Bittner wrote: >> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>> behavior. >>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>> >>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>> were >>>>>>>> doing >>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Laszlo >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>> >>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>> NullPointerException. >>>>>>>>> >>>>>>>>> Greg >>>>>>>>> >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>> the >>>>>>>>>> exception no >>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>> submitted a >>>>>>>>>>>> fix. >>>>>>>>>>>> >>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>> >>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>> the >>>>>>>>>>>> quotes) in >>>>>>>>>>>> the >>>>>>>>>>>> search bar. >>>>>>>>>>>> >>>>>>>>>>>> Greg >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>> >>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>> are in >>>>>>>>>>>>> your >>>>>>>>>>>>> DB? >>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>> or >>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>> >>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> Laszlo >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>> , >>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>> = >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>> , >>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>> = >>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>> problem >>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>> nothing >>>>>>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>> great >>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>> bug.). >>>>>>> Because those bugs were irritating, we switched to >>>>>>> nightly. Still >>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>> installed >>>>>>> new >>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>> libvirtd) >>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>> virtualized engine works great and I delete old one. Next >>>>>>> day I >>>>>>> came >>>>>>> to work and engine web page was very very slow, every >>>>>>> operation >>>>>>> takes >>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>> converted >>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>> because of >>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>> logs and >>>>>>> found this problem, which I posted in first email (VMs still >>>>>>> running, >>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>> number >>>>>>> of >>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>> VMs >>>>>>> are >>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>> may be my >>>>>>> fault during recovery). >>>>>>> >>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>> engine >>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>> Sorry. >>>>>>> >>>>>>> Thank you, >>>>>>> Jakub Bittner, NSA Czech >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>> are >>>>>> gone >>>>>> again. >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> can you verify the db from before the upgrade has the VMs? >>>> I dont know what you mean by verify database. Please, could you >>>> point me >>>> how to do it? Thank you >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> I assume from your expectation to see the VMs from the previous >>> install that you upgraded your system rather than do a clean >>> install? >>> in an upgrade, there should be a backup of the db from before the >>> upgrade. >>> (in a clean install, engine is not aware of VMs running on >>> hosts from >>> previous installs - we need to cover "import existing storage >>> domain" >>> in a future version to close this gap. current alternatives >>> around >>> this are to convert the storage domain to an export domain >>> (for an >>> nfs >>> data domain) an re-import them to the system or a few other >>> options. >>> >>> how many VMs are there? >> They(VMs) are deleted. So, my situation is this: >> >> I have clean install of ovirt 3.3.0beta with imported VMs from >> export >> domain. Everything works except snapshots. But when I restart >> ovirt >> mgmt >> server, VMs will disappear. Right now I am doing hourly backup >> of DB. > this is very fishy... > can you arrange for a remote access session for someone to take > a look? > >> I can not use workaround with converting domain to export domain, >> because we use iscsi. It would be perfect if I can retake existing >> domain. Another gap is if I dont detach export domain I can not >> attach >> it to new datacenter and I have to manually edit some files. It >> would >> be >> absolutely perfect to do it from ovirt web gui (for example by >> "force >> override button") > i thought 3.3 allowed for that, but i could be wrong. Well, I will backup all VMs to export domain, than create snapshot of ovirt and I will try to restart that server if its going to delete VMs from DB. Than I will report back. I think, that remote session will not be possible, but I can be wrong (I hope).
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello, right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
On 08/14/2013 05:19 PM, Jakub Bittner wrote:
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a):
Ahoj Jakub,
Just one more idea: could you turn statement logging on in postgresql before you try again? It is in /var/lib/pgsql/data/postgresql.conf log_statement = 'all'
----- Original Message ----- > From: "Jakub Bittner" <j.bittner@nbu.cz> > To: "Itamar Heim" <iheim@redhat.com> > Cc: users@ovirt.org > Sent: Tuesday, August 13, 2013 5:20:29 PM > Subject: Re: [Users] All VMs disappeared > > Dne 13.8.2013 17:02, Itamar Heim napsal(a): >> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>> behavior. >>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>> >>>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>>> were >>>>>>>>> doing >>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Laszlo >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>> >>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>> NullPointerException. >>>>>>>>>> >>>>>>>>>> Greg >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>>> the >>>>>>>>>>> exception no >>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Laszlo >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>>> submitted a >>>>>>>>>>>>> fix. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>> >>>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>>> the >>>>>>>>>>>>> quotes) in >>>>>>>>>>>>> the >>>>>>>>>>>>> search bar. >>>>>>>>>>>>> >>>>>>>>>>>>> Greg >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>>> are in >>>>>>>>>>>>>> your >>>>>>>>>>>>>> DB? >>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>> or >>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>> >>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>>> nothing >>>>>>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>>> great >>>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>>> bug.). >>>>>>>> Because those bugs were irritating, we switched to >>>>>>>> nightly. Still >>>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>>> installed >>>>>>>> new >>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>> libvirtd) >>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>> virtualized engine works great and I delete old one. Next >>>>>>>> day I >>>>>>>> came >>>>>>>> to work and engine web page was very very slow, every >>>>>>>> operation >>>>>>>> takes >>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>> converted >>>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>>> because of >>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>> logs and >>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>> running, >>>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>>> number >>>>>>>> of >>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>>> VMs >>>>>>>> are >>>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>>> may be my >>>>>>>> fault during recovery). >>>>>>>> >>>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>>> engine >>>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>>> Sorry. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Jakub Bittner, NSA Czech >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>>> are >>>>>>> gone >>>>>>> again. >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> can you verify the db from before the upgrade has the VMs? >>>>> I dont know what you mean by verify database. Please, could you >>>>> point me >>>>> how to do it? Thank you >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> I assume from your expectation to see the VMs from the previous >>>> install that you upgraded your system rather than do a clean >>>> install? >>>> in an upgrade, there should be a backup of the db from before the >>>> upgrade. >>>> (in a clean install, engine is not aware of VMs running on >>>> hosts from >>>> previous installs - we need to cover "import existing storage >>>> domain" >>>> in a future version to close this gap. current alternatives >>>> around >>>> this are to convert the storage domain to an export domain >>>> (for an >>>> nfs >>>> data domain) an re-import them to the system or a few other >>>> options. >>>> >>>> how many VMs are there? >>> They(VMs) are deleted. So, my situation is this: >>> >>> I have clean install of ovirt 3.3.0beta with imported VMs from >>> export >>> domain. Everything works except snapshots. But when I restart >>> ovirt >>> mgmt >>> server, VMs will disappear. Right now I am doing hourly backup >>> of DB. >> this is very fishy... >> can you arrange for a remote access session for someone to take >> a look? >> >>> I can not use workaround with converting domain to export domain, >>> because we use iscsi. It would be perfect if I can retake existing >>> domain. Another gap is if I dont detach export domain I can not >>> attach >>> it to new datacenter and I have to manually edit some files. It >>> would >>> be >>> absolutely perfect to do it from ovirt web gui (for example by >>> "force >>> override button") >> i thought 3.3 allowed for that, but i could be wrong. > Well, I will backup all VMs to export domain, than create > snapshot of > ovirt and I will try to restart that server if its going to > delete VMs > from DB. Than I will report back. I think, that remote session > will not > be possible, but I can be wrong (I hope). > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello,
right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
Laszlo hinted this may be around some compnesation logic - detecting an incomplete import, thus deleting them. yair - what should the logs indicate for this?
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: "Michal Skrivanek" <mskrivan@redhat.com>, users@ovirt.org Sent: Wednesday, August 14, 2013 5:27:17 PM Subject: Re: [Users] All VMs disappeared
On 08/14/2013 05:19 PM, Jakub Bittner wrote:
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): > Ahoj Jakub, > > Just one more idea: could you turn statement logging on in postgresql > before you try again? > It is in /var/lib/pgsql/data/postgresql.conf > log_statement = 'all' > > > ----- Original Message ----- >> From: "Jakub Bittner" <j.bittner@nbu.cz> >> To: "Itamar Heim" <iheim@redhat.com> >> Cc: users@ovirt.org >> Sent: Tuesday, August 13, 2013 5:20:29 PM >> Subject: Re: [Users] All VMs disappeared >> >> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>>> behavior. >>>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>>> >>>>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>>>> were >>>>>>>>>> doing >>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>>> NullPointerException. >>>>>>>>>>> >>>>>>>>>>> Greg >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>>>> the >>>>>>>>>>>> exception no >>>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Laszlo >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>>>> submitted a >>>>>>>>>>>>>> fix. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>>>> the >>>>>>>>>>>>>> quotes) in >>>>>>>>>>>>>> the >>>>>>>>>>>>>> search bar. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Greg >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>>>> are in >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> DB? >>>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>>>> nothing >>>>>>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>>>> great >>>>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>>>> bug.). >>>>>>>>> Because those bugs were irritating, we switched to >>>>>>>>> nightly. Still >>>>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>>>> installed >>>>>>>>> new >>>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>>> libvirtd) >>>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>>> virtualized engine works great and I delete old one. Next >>>>>>>>> day I >>>>>>>>> came >>>>>>>>> to work and engine web page was very very slow, every >>>>>>>>> operation >>>>>>>>> takes >>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>>> converted >>>>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>>>> because of >>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>>> logs and >>>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>>> running, >>>>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>>>> number >>>>>>>>> of >>>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>>>> VMs >>>>>>>>> are >>>>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>>>> may be my >>>>>>>>> fault during recovery). >>>>>>>>> >>>>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>>>> engine >>>>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>>>> Sorry. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Jakub Bittner, NSA Czech >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>>>> are >>>>>>>> gone >>>>>>>> again. >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> can you verify the db from before the upgrade has the VMs? >>>>>> I dont know what you mean by verify database. Please, could you >>>>>> point me >>>>>> how to do it? Thank you >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> I assume from your expectation to see the VMs from the previous >>>>> install that you upgraded your system rather than do a clean >>>>> install? >>>>> in an upgrade, there should be a backup of the db from before the >>>>> upgrade. >>>>> (in a clean install, engine is not aware of VMs running on >>>>> hosts from >>>>> previous installs - we need to cover "import existing storage >>>>> domain" >>>>> in a future version to close this gap. current alternatives >>>>> around >>>>> this are to convert the storage domain to an export domain >>>>> (for an >>>>> nfs >>>>> data domain) an re-import them to the system or a few other >>>>> options. >>>>> >>>>> how many VMs are there? >>>> They(VMs) are deleted. So, my situation is this: >>>> >>>> I have clean install of ovirt 3.3.0beta with imported VMs from >>>> export >>>> domain. Everything works except snapshots. But when I restart >>>> ovirt >>>> mgmt >>>> server, VMs will disappear. Right now I am doing hourly backup >>>> of DB. >>> this is very fishy... >>> can you arrange for a remote access session for someone to take >>> a look? >>> >>>> I can not use workaround with converting domain to export domain, >>>> because we use iscsi. It would be perfect if I can retake existing >>>> domain. Another gap is if I dont detach export domain I can not >>>> attach >>>> it to new datacenter and I have to manually edit some files. It >>>> would >>>> be >>>> absolutely perfect to do it from ovirt web gui (for example by >>>> "force >>>> override button") >>> i thought 3.3 allowed for that, but i could be wrong. >> Well, I will backup all VMs to export domain, than create >> snapshot of >> ovirt and I will try to restart that server if its going to >> delete VMs >> from DB. Than I will report back. I think, that remote session >> will not >> be possible, but I can be wrong (I hope). >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello,
right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
Laszlo hinted this may be around some compnesation logic - detecting an incomplete import, thus deleting them. yair - what should the logs indicate for this?
There's a possible issue which might have caused it - after the latest async task manager changed - on some of the commands that create tasks, the parameters info is being stored in the db before it's being overriden by the command when the tasks are created (for example - the command may set new ids for the entities as done in the import command CanDoAction method) - on a failed import the engine attempts to revert the operation..if the parameters stored data isn't correct, other object may be deleted. Yair, if it's not the compensation - this might be the case.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: "Michal Skrivanek" <mskrivan@redhat.com>, users@ovirt.org Sent: Wednesday, August 14, 2013 4:27:17 PM Subject: Re: [Users] All VMs disappeared
On 08/14/2013 05:19 PM, Jakub Bittner wrote:
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 14, 2013 8:09:45 AM Subject: Re: [Users] All VMs disappeared
Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): > Ahoj Jakub, > > Just one more idea: could you turn statement logging on in postgresql > before you try again? > It is in /var/lib/pgsql/data/postgresql.conf > log_statement = 'all' > > > ----- Original Message ----- >> From: "Jakub Bittner" <j.bittner@nbu.cz> >> To: "Itamar Heim" <iheim@redhat.com> >> Cc: users@ovirt.org >> Sent: Tuesday, August 13, 2013 5:20:29 PM >> Subject: Re: [Users] All VMs disappeared >> >> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>>> behavior. >>>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>>> >>>>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>>>> were >>>>>>>>>> doing >>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Laszlo >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>> >>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>>> NullPointerException. >>>>>>>>>>> >>>>>>>>>>> Greg >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>>>> the >>>>>>>>>>>> exception no >>>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Laszlo >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>>>> submitted a >>>>>>>>>>>>>> fix. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>>>> the >>>>>>>>>>>>>> quotes) in >>>>>>>>>>>>>> the >>>>>>>>>>>>>> search bar. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Greg >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>>>> are in >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> DB? >>>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>>>> nothing >>>>>>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>>>> great >>>>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>>>> bug.). >>>>>>>>> Because those bugs were irritating, we switched to >>>>>>>>> nightly. Still >>>>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>>>> installed >>>>>>>>> new >>>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>>> libvirtd) >>>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>>> virtualized engine works great and I delete old one. Next >>>>>>>>> day I >>>>>>>>> came >>>>>>>>> to work and engine web page was very very slow, every >>>>>>>>> operation >>>>>>>>> takes >>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>>> converted >>>>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>>>> because of >>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>>> logs and >>>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>>> running, >>>>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>>>> number >>>>>>>>> of >>>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>>>> VMs >>>>>>>>> are >>>>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>>>> may be my >>>>>>>>> fault during recovery). >>>>>>>>> >>>>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>>>> engine >>>>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>>>> Sorry. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Jakub Bittner, NSA Czech >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>>>> are >>>>>>>> gone >>>>>>>> again. >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> can you verify the db from before the upgrade has the VMs? >>>>>> I dont know what you mean by verify database. Please, could you >>>>>> point me >>>>>> how to do it? Thank you >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> I assume from your expectation to see the VMs from the previous >>>>> install that you upgraded your system rather than do a clean >>>>> install? >>>>> in an upgrade, there should be a backup of the db from before the >>>>> upgrade. >>>>> (in a clean install, engine is not aware of VMs running on >>>>> hosts from >>>>> previous installs - we need to cover "import existing storage >>>>> domain" >>>>> in a future version to close this gap. current alternatives >>>>> around >>>>> this are to convert the storage domain to an export domain >>>>> (for an >>>>> nfs >>>>> data domain) an re-import them to the system or a few other >>>>> options. >>>>> >>>>> how many VMs are there? >>>> They(VMs) are deleted. So, my situation is this: >>>> >>>> I have clean install of ovirt 3.3.0beta with imported VMs from >>>> export >>>> domain. Everything works except snapshots. But when I restart >>>> ovirt >>>> mgmt >>>> server, VMs will disappear. Right now I am doing hourly backup >>>> of DB. >>> this is very fishy... >>> can you arrange for a remote access session for someone to take >>> a look? >>> >>>> I can not use workaround with converting domain to export domain, >>>> because we use iscsi. It would be perfect if I can retake existing >>>> domain. Another gap is if I dont detach export domain I can not >>>> attach >>>> it to new datacenter and I have to manually edit some files. It >>>> would >>>> be >>>> absolutely perfect to do it from ovirt web gui (for example by >>>> "force >>>> override button") >>> i thought 3.3 allowed for that, but i could be wrong. >> Well, I will backup all VMs to export domain, than create >> snapshot of >> ovirt and I will try to restart that server if its going to >> delete VMs >> from DB. Than I will report back. I think, that remote session >> will not >> be possible, but I can be wrong (I hope). >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello,
right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
Laszlo hinted this may be around some compnesation logic - detecting an incomplete import, thus deleting them. yair - what should the logs indicate for this?
I thought of the compensation because it runs after engine start and does changes in the DB if there is something to compensate in business_entity_snapshot. So for example having records there could undo the import of VM's and at the same time delete their disks from the storage. However, when last time I had anything to do with the compensation, if it failed, it did not allow the engine to start. I don't know if it is still like that, I just noticed last week that some services may fail to configure and the engine ignores them in such cases. Jakub, do you have anything in the business_entity_snapshot table?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Cituji "Laszlo Hornyak" <lhornyak@redhat.com>:
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: "Michal Skrivanek" <mskrivan@redhat.com>, users@ovirt.org Sent: Wednesday, August 14, 2013 4:27:17 PM Subject: Re: [Users] All VMs disappeared
On 08/14/2013 05:19 PM, Jakub Bittner wrote:
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message -----
From: "Laszlo Hornyak" <lhornyak@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: users@ovirt.org Sent: Wednesday, August 14, 2013 10:39:48 AM Subject: Re: [Users] All VMs disappeared
----- Original Message ----- > From: "Jakub Bittner" <j.bittner@nbu.cz> > To: users@ovirt.org > Sent: Wednesday, August 14, 2013 8:09:45 AM > Subject: Re: [Users] All VMs disappeared > > Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): >> Ahoj Jakub, >> >> Just one more idea: could you turn statement logging on in postgresql >> before you try again? >> It is in /var/lib/pgsql/data/postgresql.conf >> log_statement = 'all' >> >> >> ----- Original Message ----- >>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>> To: "Itamar Heim" <iheim@redhat.com> >>> Cc: users@ovirt.org >>> Sent: Tuesday, August 13, 2013 5:20:29 PM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >>>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>>>> behavior. >>>>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>>>> >>>>>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>>>>> were >>>>>>>>>>> doing >>>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Laszlo >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>> >>>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>>>> NullPointerException. >>>>>>>>>>>> >>>>>>>>>>>> Greg >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>>>>> the >>>>>>>>>>>>> exception no >>>>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> Laszlo >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>>>>> submitted a >>>>>>>>>>>>>>> fix. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> quotes) in >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> search bar. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Greg >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>>>>> are in >>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>> DB? >>>>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>>>>> nothing >>>>>>>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>>>>> great >>>>>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>>>>> bug.). >>>>>>>>>> Because those bugs were irritating, we switched to >>>>>>>>>> nightly. Still >>>>>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>>>>> installed >>>>>>>>>> new >>>>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>>>> libvirtd) >>>>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>>>> virtualized engine works great and I delete old one. Next >>>>>>>>>> day I >>>>>>>>>> came >>>>>>>>>> to work and engine web page was very very slow, every >>>>>>>>>> operation >>>>>>>>>> takes >>>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>>>> converted >>>>>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>>>>> because of >>>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>>>> logs and >>>>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>>>> running, >>>>>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>>>>> number >>>>>>>>>> of >>>>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>>>>> VMs >>>>>>>>>> are >>>>>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>>>>> may be my >>>>>>>>>> fault during recovery). >>>>>>>>>> >>>>>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>>>>> engine >>>>>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>>>>> Sorry. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Jakub Bittner, NSA Czech >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>>>>> are >>>>>>>>> gone >>>>>>>>> again. >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> can you verify the db from before the upgrade has the VMs? >>>>>>> I dont know what you mean by verify database. Please, could you >>>>>>> point me >>>>>>> how to do it? Thank you >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> I assume from your expectation to see the VMs from the previous >>>>>> install that you upgraded your system rather than do a clean >>>>>> install? >>>>>> in an upgrade, there should be a backup of the db from before the >>>>>> upgrade. >>>>>> (in a clean install, engine is not aware of VMs running on >>>>>> hosts from >>>>>> previous installs - we need to cover "import existing storage >>>>>> domain" >>>>>> in a future version to close this gap. current alternatives >>>>>> around >>>>>> this are to convert the storage domain to an export domain >>>>>> (for an >>>>>> nfs >>>>>> data domain) an re-import them to the system or a few other >>>>>> options. >>>>>> >>>>>> how many VMs are there? >>>>> They(VMs) are deleted. So, my situation is this: >>>>> >>>>> I have clean install of ovirt 3.3.0beta with imported VMs from >>>>> export >>>>> domain. Everything works except snapshots. But when I restart >>>>> ovirt >>>>> mgmt >>>>> server, VMs will disappear. Right now I am doing hourly backup >>>>> of DB. >>>> this is very fishy... >>>> can you arrange for a remote access session for someone to take >>>> a look? >>>> >>>>> I can not use workaround with converting domain to export domain, >>>>> because we use iscsi. It would be perfect if I can retake existing >>>>> domain. Another gap is if I dont detach export domain I can not >>>>> attach >>>>> it to new datacenter and I have to manually edit some files. It >>>>> would >>>>> be >>>>> absolutely perfect to do it from ovirt web gui (for example by >>>>> "force >>>>> override button") >>>> i thought 3.3 allowed for that, but i could be wrong. >>> Well, I will backup all VMs to export domain, than create >>> snapshot of >>> ovirt and I will try to restart that server if its going to >>> delete VMs >>> from DB. Than I will report back. I think, that remote session >>> will not >>> be possible, but I can be wrong (I hope). >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> > I change it. But I can not restart postgres. You can restart it only when the engine is down, it will not restart with database connections open.
> _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello,
right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
Laszlo hinted this may be around some compnesation logic - detecting an incomplete import, thus deleting them. yair - what should the logs indicate for this?
I thought of the compensation because it runs after engine start and does changes in the DB if there is something to compensate in business_entity_snapshot. So for example having records there could undo the import of VM's and at the same time delete their disks from the storage. However, when last time I had anything to do with the compensation, if it failed, it did not allow the engine to start. I don't know if it is still like that, I just noticed last week that some services may fail to configure and the engine ignores them in such cases.
Jakub, do you have anything in the business_entity_snapshot table?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I do appologies, but I will be at work on Friday 23 and than I take a look. Sorry. But thank you for hints ;-) ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
Dne 14.8.2013 18:04, j.bittner@nbu.cz napsal(a):
Cituji "Laszlo Hornyak" <lhornyak@redhat.com>:
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: "Michal Skrivanek" <mskrivan@redhat.com>, users@ovirt.org Sent: Wednesday, August 14, 2013 4:27:17 PM Subject: Re: [Users] All VMs disappeared
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a):
Jakub, are only your vms disappear? what about the disks?
----- Original Message ----- > From: "Laszlo Hornyak" <lhornyak@redhat.com> > To: "Jakub Bittner" <j.bittner@nbu.cz> > Cc: users@ovirt.org > Sent: Wednesday, August 14, 2013 10:39:48 AM > Subject: Re: [Users] All VMs disappeared > > > > ----- Original Message ----- >> From: "Jakub Bittner" <j.bittner@nbu.cz> >> To: users@ovirt.org >> Sent: Wednesday, August 14, 2013 8:09:45 AM >> Subject: Re: [Users] All VMs disappeared >> >> Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): >>> Ahoj Jakub, >>> >>> Just one more idea: could you turn statement logging on in
On 08/14/2013 05:19 PM, Jakub Bittner wrote: postgresql
>>> before you try again? >>> It is in /var/lib/pgsql/data/postgresql.conf >>> log_statement = 'all' >>> >>> >>> ----- Original Message ----- >>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>> To: "Itamar Heim" <iheim@redhat.com> >>>> Cc: users@ovirt.org >>>> Sent: Tuesday, August 13, 2013 5:20:29 PM >>>> Subject: Re: [Users] All VMs disappeared >>>> >>>> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >>>>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>>>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>>>>> behavior. >>>>>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>>>>> >>>>>>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>>>>>> were >>>>>>>>>>>> doing >>>>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Laszlo >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>> >>>>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>>>>> NullPointerException. >>>>>>>>>>>>> >>>>>>>>>>>>> Greg >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>> >>>>>>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>>>>>> the >>>>>>>>>>>>>> exception no >>>>>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>>>>>> submitted a >>>>>>>>>>>>>>>> fix. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> quotes) in >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> search bar. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Greg >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>>>>>> are in >>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>> DB? >>>>>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>>>>>> nothing >>>>>>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>>>>>> great >>>>>>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>>>>>> bug.). >>>>>>>>>>> Because those bugs were irritating, we switched to >>>>>>>>>>> nightly. Still >>>>>>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>>>>>> installed >>>>>>>>>>> new >>>>>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>>>>> libvirtd) >>>>>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>>>>> virtualized engine works great and I delete old one. Next >>>>>>>>>>> day I >>>>>>>>>>> came >>>>>>>>>>> to work and engine web page was very very slow, every >>>>>>>>>>> operation >>>>>>>>>>> takes >>>>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>>>>> converted >>>>>>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>>>>>> because of >>>>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>>>>> logs and >>>>>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>>>>> running, >>>>>>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>>>>>> number >>>>>>>>>>> of >>>>>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>>>>>> VMs >>>>>>>>>>> are >>>>>>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>>>>>> may be my >>>>>>>>>>> fault during recovery). >>>>>>>>>>> >>>>>>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>>>>>> engine >>>>>>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>>>>>> Sorry. >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Jakub Bittner, NSA Czech >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>>>>>> are >>>>>>>>>> gone >>>>>>>>>> again. >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> can you verify the db from before the upgrade has the VMs? >>>>>>>> I dont know what you mean by verify database. Please, could you >>>>>>>> point me >>>>>>>> how to do it? Thank you >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> I assume from your expectation to see the VMs from the previous >>>>>>> install that you upgraded your system rather than do a clean >>>>>>> install? >>>>>>> in an upgrade, there should be a backup of the db from before the >>>>>>> upgrade. >>>>>>> (in a clean install, engine is not aware of VMs running on >>>>>>> hosts from >>>>>>> previous installs - we need to cover "import existing storage >>>>>>> domain" >>>>>>> in a future version to close this gap. current alternatives >>>>>>> around >>>>>>> this are to convert the storage domain to an export domain >>>>>>> (for an >>>>>>> nfs >>>>>>> data domain) an re-import them to the system or a few other >>>>>>> options. >>>>>>> >>>>>>> how many VMs are there? >>>>>> They(VMs) are deleted. So, my situation is this: >>>>>> >>>>>> I have clean install of ovirt 3.3.0beta with imported VMs from >>>>>> export >>>>>> domain. Everything works except snapshots. But when I restart >>>>>> ovirt >>>>>> mgmt >>>>>> server, VMs will disappear. Right now I am doing hourly backup >>>>>> of DB. >>>>> this is very fishy... >>>>> can you arrange for a remote access session for someone to take >>>>> a look? >>>>> >>>>>> I can not use workaround with converting domain to export domain, >>>>>> because we use iscsi. It would be perfect if I can retake existing >>>>>> domain. Another gap is if I dont detach export domain I can not >>>>>> attach >>>>>> it to new datacenter and I have to manually edit some files. It >>>>>> would >>>>>> be >>>>>> absolutely perfect to do it from ovirt web gui (for example by >>>>>> "force >>>>>> override button") >>>>> i thought 3.3 allowed for that, but i could be wrong. >>>> Well, I will backup all VMs to export domain, than create >>>> snapshot of >>>> ovirt and I will try to restart that server if its going to >>>> delete VMs >>>> from DB. Than I will report back. I think, that remote session >>>> will not >>>> be possible, but I can be wrong (I hope). >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >> I change it. But I can not restart postgres. > You can restart it only when the engine is down, it will not restart > with > database connections open. > >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back.
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello,
right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
Laszlo hinted this may be around some compnesation logic - detecting an incomplete import, thus deleting them. yair - what should the logs indicate for this?
I thought of the compensation because it runs after engine start and does changes in the DB if there is something to compensate in business_entity_snapshot. So for example having records there could undo the import of VM's and at the same time delete their disks from the storage. However, when last time I had anything to do with the compensation, if it failed, it did not allow the engine to start. I don't know if it is still like that, I just noticed last week that some services may fail to configure and the engine ignores them in such cases.
Jakub, do you have anything in the business_entity_snapshot table?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I do appologies, but I will be at work on Friday 23 and than I take a look. Sorry. But thank you for hints ;-)
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
May I ask where to find engine db password or how to get to pg?
Dne 23.8.2013 10:24, Jakub Bittner napsal(a):
Dne 14.8.2013 18:04, j.bittner@nbu.cz napsal(a):
Cituji "Laszlo Hornyak" <lhornyak@redhat.com>:
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Jakub Bittner" <j.bittner@nbu.cz> Cc: "Michal Skrivanek" <mskrivan@redhat.com>, users@ovirt.org Sent: Wednesday, August 14, 2013 4:27:17 PM Subject: Re: [Users] All VMs disappeared
Dne 14.8.2013 10:48, Jakub Bittner napsal(a):
Dne 14.8.2013 09:57, Liron Aravot napsal(a): > Jakub, are only your vms disappear? what about the disks? > > ----- Original Message ----- >> From: "Laszlo Hornyak" <lhornyak@redhat.com> >> To: "Jakub Bittner" <j.bittner@nbu.cz> >> Cc: users@ovirt.org >> Sent: Wednesday, August 14, 2013 10:39:48 AM >> Subject: Re: [Users] All VMs disappeared >> >> >> >> ----- Original Message ----- >>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>> To: users@ovirt.org >>> Sent: Wednesday, August 14, 2013 8:09:45 AM >>> Subject: Re: [Users] All VMs disappeared >>> >>> Dne 13.8.2013 18:29, Laszlo Hornyak napsal(a): >>>> Ahoj Jakub, >>>> >>>> Just one more idea: could you turn statement logging on in
On 08/14/2013 05:19 PM, Jakub Bittner wrote: postgresql
>>>> before you try again? >>>> It is in /var/lib/pgsql/data/postgresql.conf >>>> log_statement = 'all' >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>> To: "Itamar Heim" <iheim@redhat.com> >>>>> Cc: users@ovirt.org >>>>> Sent: Tuesday, August 13, 2013 5:20:29 PM >>>>> Subject: Re: [Users] All VMs disappeared >>>>> >>>>> Dne 13.8.2013 17:02, Itamar Heim napsal(a): >>>>>> On 08/13/2013 05:31 PM, Jakub Bittner wrote: >>>>>>> Dne 13.8.2013 15:39, Itamar Heim napsal(a): >>>>>>>> On 08/13/2013 08:50 AM, Jakub Bittner wrote: >>>>>>>>> Dne 12.8.2013 22:44, Itamar Heim napsal(a): >>>>>>>>>> On 08/12/2013 06:59 PM, Jakub Bittner wrote: >>>>>>>>>>> Dne 12.8.2013 14:54, Jakub Bittner napsal(a): >>>>>>>>>>>> Dne 12.8.2013 14:29, Laszlo Hornyak napsal(a): >>>>>>>>>>>>> I looked around Noam's patch and that should not cause such >>>>>>>>>>>>> behavior. >>>>>>>>>>>>> I am wondering how that lost VM's could happen. >>>>>>>>>>>>> >>>>>>>>>>>>> Jakub, can you give a more detailed description what you >>>>>>>>>>>>> were >>>>>>>>>>>>> doing >>>>>>>>>>>>> with oVirt when this happened? Maybe the bug is still there. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> Laszlo >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>>> To: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>> Cc: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>>>> <nslomian@redhat.com>, users@ovirt.org >>>>>>>>>>>>>> Sent: Monday, August 12, 2013 1:23:40 PM >>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>> >>>>>>>>>>>>>> Not the one I fixed, 987907. It was a simple UI >>>>>>>>>>>>>> NullPointerException. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Greg >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz>, "Noam Slomianko" >>>>>>>>>>>>>>> <nslomian@redhat.com>, "Greg Sheremeta" >>>>>>>>>>>>>>> <gshereme@redhat.com> >>>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>>> Sent: Monday, August 12, 2013 7:21:26 AM >>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Well if they are no longer in DB then that explains why >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> exception no >>>>>>>>>>>>>>> longer occurs, but at the cost of database corruption. >>>>>>>>>>>>>>> Noam and Greg, can these bugs cause data corruption? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>> To: "Greg Sheremeta" <gshereme@redhat.com> >>>>>>>>>>>>>>>> Cc: users@ovirt.org, "Laszlo Hornyak" >>>>>>>>>>>>>>>> <lhornyak@redhat.com> >>>>>>>>>>>>>>>> Sent: Monday, August 12, 2013 9:05:09 AM >>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dne 10.8.2013 01:54, Greg Sheremeta napsal(a): >>>>>>>>>>>>>>>>> It could also be this bug[1], for which I just >>>>>>>>>>>>>>>>> submitted a >>>>>>>>>>>>>>>>> fix. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=987907 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You can work around it by typing just "VMs:" (without >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> quotes) in >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> search bar. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Greg >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>>> From: "Laszlo Hornyak" <lhornyak@redhat.com> >>>>>>>>>>>>>>>>>> To: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>>>> Cc: users@ovirt.org >>>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 11:21:26 AM >>>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Jakub, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Could you check through DB or REST-API if the VM's >>>>>>>>>>>>>>>>>> are in >>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>> DB? >>>>>>>>>>>>>>>>>> select * from vm_static; >>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>> curl -u admin@internal:blablabla [engine-url]api/vms >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> It seems Noam fixed this issue already in >>>>>>>>>>>>>>>>>> c2295c31fa645e1ba1b94cd557bd1fecb40c8829. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>> Laszlo >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>>>> From: "Jakub Bittner" <j.bittner@nbu.cz> >>>>>>>>>>>>>>>>>>> To: users@ovirt.org >>>>>>>>>>>>>>>>>>> Sent: Friday, August 9, 2013 1:06:53 PM >>>>>>>>>>>>>>>>>>> Subject: Re: [Users] All VMs disappeared >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Dne 9.8.2013 09:48, Jakub Bittner napsal(a): >>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Iam running ovirt 3.3.0.beta1 on centos 6,4 and >>>>>>>>>>>>>>>>>>>> all our >>>>>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>>>>> disappeared from VMs tab. Nodes running on centos >>>>>>>>>>>>>>>>>>>> too. >>>>>>>>>>>>>>>>>>>> Repeating >>>>>>>>>>>>>>>>>>>> problem in log is: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2013-08-09 09:44:24,203 WARN >>>>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-45) Failed to >>>>>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>>>>> 7cb6aedf-47bc-40b0-877f-2a537fca5c64 : >>>>>>>>>>>>>>>>>>>> node2.x.com, error >>>>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2013-08-09 09:44:27,039 WARN >>>>>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-47) Failed to >>>>>>>>>>>>>>>>>>>> refresh VDS >>>>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>>>> vds = >>>>>>>>>>>>>>>>>>>> 653e34a6-6b7e-4757-9042-dc02e7a0cddf : >>>>>>>>>>>>>>>>>>>> node3.x.com, error >>>>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException, continuing.: >>>>>>>>>>>>>>>>>>>> java.lang.NullPointerException >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.proceedGuaranteedMemoryCheck(VdsUpdateRunTimeInfo.java:1313)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVmStats(VdsUpdateRunTimeInfo.java:968)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:542)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:383)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:237)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [vdsbroker.jar:] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor171.invoke(Unknown >>>>>>>>>>>>>>>>>>>> Source) [:1.7.0_25] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>>>>>>>>>> [rt.jar:1.7.0_25] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [scheduler.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [quartz.jar:] >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>> I restarted management node and both nodes and this >>>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>>> disappeared. >>>>>>>>>>>>>>>>>>> But I still can not see any VMs. >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I do not have them in database. Both commands shows >>>>>>>>>>>>>>>> nothing >>>>>>>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> we run ovirt 3.2.1 (or 2). I upgraded to RC1, everything was >>>>>>>>>>>> great >>>>>>>>>>>> (except SPICE. VNC and creating new WM due to serialization >>>>>>>>>>>> bug.). >>>>>>>>>>>> Because those bugs were irritating, we switched to >>>>>>>>>>>> nightly. Still >>>>>>>>>>>> works fine (but bug with VMs is not fixed there), than I >>>>>>>>>>>> installed >>>>>>>>>>>> new >>>>>>>>>>>> ovirt engine management server in KVM virtualization (using >>>>>>>>>>>> libvirtd) >>>>>>>>>>>> due to taking snapshots before deploying nightly builds. New >>>>>>>>>>>> virtualized engine works great and I delete old one. Next >>>>>>>>>>>> day I >>>>>>>>>>>> came >>>>>>>>>>>> to work and engine web page was very very slow, every >>>>>>>>>>>> operation >>>>>>>>>>>> takes >>>>>>>>>>>> ages, that time I found there were no VMs, no HDDs and so. I >>>>>>>>>>>> converted >>>>>>>>>>>> its qcow2 image to raw, because I think it has slow DB >>>>>>>>>>>> because of >>>>>>>>>>>> running i qcow2. Did not helped, than I take a look in nodes >>>>>>>>>>>> logs and >>>>>>>>>>>> found this problem, which I posted in first email (VMs still >>>>>>>>>>>> running, >>>>>>>>>>>> but not visible in engine VMs tab, but on host I can see >>>>>>>>>>>> number >>>>>>>>>>>> of >>>>>>>>>>>> running VMs). After restart bug disappeared, but VMs too. >>>>>>>>>>>> Today I reinstalled engine with oVirt 3.3.0rc1. But all my >>>>>>>>>>>> VMs >>>>>>>>>>>> are >>>>>>>>>>>> gone. Even from storage. (but disappearing from storage >>>>>>>>>>>> may be my >>>>>>>>>>>> fault during recovery). >>>>>>>>>>>> >>>>>>>>>>>> I can provide logs from both two nodes and (hopefully) from >>>>>>>>>>>> engine >>>>>>>>>>>> too. I can provide you with those logs, but not publicly. >>>>>>>>>>>> Sorry. >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Jakub Bittner, NSA Czech >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list >>>>>>>>>>>> Users@ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>> Well I reinstalled to 3.3.0 RC1, rebooted server and my VMs >>>>>>>>>>> are >>>>>>>>>>> gone >>>>>>>>>>> again. >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list >>>>>>>>>>> Users@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> can you verify the db from before the upgrade has the VMs? >>>>>>>>> I dont know what you mean by verify database. Please, could you >>>>>>>>> point me >>>>>>>>> how to do it? Thank you >>>>>>>>> _______________________________________________ >>>>>>>>> Users mailing list >>>>>>>>> Users@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> I assume from your expectation to see the VMs from the previous >>>>>>>> install that you upgraded your system rather than do a clean >>>>>>>> install? >>>>>>>> in an upgrade, there should be a backup of the db from before the >>>>>>>> upgrade. >>>>>>>> (in a clean install, engine is not aware of VMs running on >>>>>>>> hosts from >>>>>>>> previous installs - we need to cover "import existing storage >>>>>>>> domain" >>>>>>>> in a future version to close this gap. current alternatives >>>>>>>> around >>>>>>>> this are to convert the storage domain to an export domain >>>>>>>> (for an >>>>>>>> nfs >>>>>>>> data domain) an re-import them to the system or a few other >>>>>>>> options. >>>>>>>> >>>>>>>> how many VMs are there? >>>>>>> They(VMs) are deleted. So, my situation is this: >>>>>>> >>>>>>> I have clean install of ovirt 3.3.0beta with imported VMs from >>>>>>> export >>>>>>> domain. Everything works except snapshots. But when I restart >>>>>>> ovirt >>>>>>> mgmt >>>>>>> server, VMs will disappear. Right now I am doing hourly backup >>>>>>> of DB. >>>>>> this is very fishy... >>>>>> can you arrange for a remote access session for someone to take >>>>>> a look? >>>>>> >>>>>>> I can not use workaround with converting domain to export domain, >>>>>>> because we use iscsi. It would be perfect if I can retake existing >>>>>>> domain. Another gap is if I dont detach export domain I can not >>>>>>> attach >>>>>>> it to new datacenter and I have to manually edit some files. It >>>>>>> would >>>>>>> be >>>>>>> absolutely perfect to do it from ovirt web gui (for example by >>>>>>> "force >>>>>>> override button") >>>>>> i thought 3.3 allowed for that, but i could be wrong. >>>>> Well, I will backup all VMs to export domain, than create >>>>> snapshot of >>>>> ovirt and I will try to restart that server if its going to >>>>> delete VMs >>>>> from DB. Than I will report back. I think, that remote session >>>>> will not >>>>> be possible, but I can be wrong (I hope). >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>> I change it. But I can not restart postgres. >> You can restart it only when the engine is down, it will not restart >> with >> database connections open. >> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> I think, that disks did not disappeared, but new instance of ovirt deleted them as I thought that it can import them back. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
hello,
right now I restarted ovirt-engine and postgresql and my VMs disappeared again, but not all of them! Some of them persists. (two) And I get messages about failed export and import of those who disappeared. Than I restored postgresql from backup and everything is OK. Restarting postgresql is like using way back machine ;-)
Laszlo hinted this may be around some compnesation logic - detecting an incomplete import, thus deleting them. yair - what should the logs indicate for this?
I thought of the compensation because it runs after engine start and does changes in the DB if there is something to compensate in business_entity_snapshot. So for example having records there could undo the import of VM's and at the same time delete their disks from the storage. However, when last time I had anything to do with the compensation, if it failed, it did not allow the engine to start. I don't know if it is still like that, I just noticed last week that some services may fail to configure and the engine ignores them in such cases.
Jakub, do you have anything in the business_entity_snapshot table?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I do appologies, but I will be at work on Friday 23 and than I take a look. Sorry. But thank you for hints ;-)
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
May I ask where to find engine db password or how to get to pg? _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
My business_entity_snapshot is empty.
On 09.08.2013 09:48, Jakub Bittner wrote:
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
I have the same problem, but with a slightly different setup: - fresh install of Fedora 19 (all updates installed) - fresh install of oVirt 3.3 RC (no upgrade) - All-in-One setup with only local storage - VMs installed from scratch (no import) - VM/node/engine uptime was about 22 hours before reboot Now the problem is very the same as reported by Jakub: Every time I reboot ovirt-engine the VMs will disappear. No matter what I do before rebooting ovirt-engine - set storage to maintenance, set host to maintenance - it's always the same. One interesting thing to note: Only VMs created from my custom template will disappear! If I create a VM from the "blank" template, it will NOT disappear. Of course, the VM Disks can still be found in the filesystem, but it's a nightmare trying to guess what disk should be attached to what VM... I consider these VMs lost, since it's (fortunately) my test server, but still a day of work is lost. Here are some interesting log entries from engine startup AFTER REBOOT: 2013-08-26 23:03:20,479 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-6-thread-7) Failing task with empty vdsm id AsyncTaskType copyImage : Task 7fa218c7-14c3-4a7f-97e8-341f18c6cb3d Parent Command AddVmFromTemplate 2013-08-26 23:03:20,497 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-6-thread-7) Calling updateTask for task with out vdsm id and AsyncTaskType copyImage : Task 7fa218c7-14c3-4a7f-97e8-341f18c6cb3d Parent Command AddVmFromTemplate Parameters class {3} 2013-08-26 23:03:20,506 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 4202990d-e85f-4935-a91b-a832bc8582cc 2013-08-26 23:03:20,787 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (pool-6-thread-7) [10025b02] Lock Acquired to object EngineLock [exclusiveLocks= key: vm1.example.com value: VM_NAME 2013-08-26 23:03:20,788 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-6-thread-7) [10025b02] CommandMultiAsyncTasks::AttachTask: Attaching task 00000000-0000-0000-0000-000000000000 to command 4202990d-e85f-4935-a91b-a832bc8582cc. 2013-08-26 23:03:20,796 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-6-thread-7) [10025b02] BaseAsyncTask::LogEndTaskFailure: Task 00000000-0000-0000-0000-000000000000 (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure: 2013-08-26 23:03:20,805 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-6-thread-7) [10025b02] task with DB task ID 00000000-0000-0000-0000-000000000000 has empty vdsm task ID and is about to be cleared 2013-08-26 23:03:20,806 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) [10025b02] CommandAsyncTask::EndActionIfNecessary: All tasks of command 4202990d-e85f-4935-a91b-a832bc8582cc has ended -> executing EndAction 2013-08-26 23:03:20,807 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) [10025b02] CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 4202990d-e85f-4935-a91b-a832bc8582cc): calling EndAction . Needless to say that I'm ABSOLUTELY SURE the template-copy tasks actually did succeed before rebooting, as I was using the VMs, installing software, rebooting, ... they were all running fine for 22 hours! These messages exist for every missing VM. A show-stopper for 3.3 if you ask me :-/ Thanks - Frank
On 26.08.2013 23:40, Frank Wall wrote:
Every time I reboot ovirt-engine the VMs will disappear. No matter what I do before rebooting ovirt-engine - set storage to maintenance, set host to maintenance - it's always the same.
On IRC #ovirt I was asked to have a look at the "async_tasks" table while cloning a VM from a template. An entry is created in this table, but the vdsm_id is incorrect: engine=# select task_id,action_type,status,vdsm_task_id from async_tasks; 6fa81c23-6f64-444c-bab0-f178fa9233d5 | 2 | 2 | 00000000-0000-0000-0000-000000000000 Maybe this empty vdsm_id is related to the disappearing vm issue? Thanks - Frank
Hi Frank, The vdsm_task_id with empty guid should be legal in ovirt. What's more there is a javadoc for the case /** * If a command is partially submitted to vdsm, the empty place holders * can be removed from the database and we poll for the completion of tasks * that were already submitted. Once the tasks that were submitted finish * execution, they are marked to be failed by adding them to the * partiallyCompletedCommandTasks list. * If none of the tasks were submitted to vdsm, the empty place holders * are deleted from the database and we endAction on the command with * failure If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up. I need to look into this but I think it should be safe to remove this record while the engine is down. Make sure to make a backup if you try! Laszlo ----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 1:47:28 PM Subject: Re: [Users] All VMs disappeared
On 26.08.2013 23:40, Frank Wall wrote:
Every time I reboot ovirt-engine the VMs will disappear. No matter what I do before rebooting ovirt-engine - set storage to maintenance, set host to maintenance - it's always the same.
On IRC #ovirt I was asked to have a look at the "async_tasks" table while cloning a VM from a template. An entry is created in this table, but the vdsm_id is incorrect:
engine=# select task_id,action_type,status,vdsm_task_id from async_tasks; 6fa81c23-6f64-444c-bab0-f178fa9233d5 | 2 | 2 | 00000000-0000-0000-0000-000000000000
Maybe this empty vdsm_id is related to the disappearing vm issue?
Thanks - Frank _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Laszlo, On 27.08.2013 14:24, Laszlo Hornyak wrote:
If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up.
just to be sure, please note, that the task actually completed successfully from my point of view. The new VM (direved from the template clone task) is fully usable and on the webadmin the task is shown as "completed". So maybe at some point in time the engine does get the job status wrong and the problem begins...
I need to look into this but I think it should be safe to remove this record while the engine is down. Make sure to make a backup if you try!
OK, I did the following test: - created a new VM from my custom template (with cloned Disk) - waited for the job to complete - manually deleted the related DB entry from the table "async_tasks" - set host/node to maintenance - rebootet engine Result: The VM did NOT disappear! :-) I think somehow ovirt-engine does get the results of the job wrong, thus it does not recognize that the job completed *successfully* and does a retry after reboot (which fails horribly and leads to disappearing VMs). Thanks - Frank
Thank you Frank, I will investigate this issue. ----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 2:44:44 PM Subject: Re: [Users] All VMs disappeared
Hi Laszlo,
On 27.08.2013 14:24, Laszlo Hornyak wrote:
If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up.
just to be sure, please note, that the task actually completed successfully from my point of view. The new VM (direved from the template clone task) is fully usable and on the webadmin the task is shown as "completed".
So maybe at some point in time the engine does get the job status wrong and the problem begins...
I need to look into this but I think it should be safe to remove this record while the engine is down. Make sure to make a backup if you try!
OK, I did the following test:
- created a new VM from my custom template (with cloned Disk) - waited for the job to complete - manually deleted the related DB entry from the table "async_tasks" - set host/node to maintenance - rebootet engine
Result: The VM did NOT disappear! :-)
I think somehow ovirt-engine does get the results of the job wrong, thus it does not recognize that the job completed *successfully* and does a retry after reboot (which fails horribly and leads to disappearing VMs).
Thanks - Frank
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Dne 27.8.2013 16:01, Laszlo Hornyak napsal(a):
Thank you Frank, I will investigate this issue.
----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 2:44:44 PM Subject: Re: [Users] All VMs disappeared
Hi Laszlo,
On 27.08.2013 14:24, Laszlo Hornyak wrote:
If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up. just to be sure, please note, that the task actually completed successfully from my point of view. The new VM (direved from the template clone task) is fully usable and on the webadmin the task is shown as "completed".
So maybe at some point in time the engine does get the job status wrong and the problem begins...
I need to look into this but I think it should be safe to remove this record while the engine is down. Make sure to make a backup if you try! OK, I did the following test:
- created a new VM from my custom template (with cloned Disk) - waited for the job to complete - manually deleted the related DB entry from the table "async_tasks" - set host/node to maintenance - rebootet engine
Result: The VM did NOT disappear! :-)
I think somehow ovirt-engine does get the results of the job wrong, thus it does not recognize that the job completed *successfully* and does a retry after reboot (which fails horribly and leads to disappearing VMs).
Thanks - Frank
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users Hi,
I manually deleted async_tasks and I dont have problem with disappearing of VMs. Thanks
Liron, looks like you were right and it was not the business_entities_snapshot but the async_tasks that caused the data loss. I hope there are no more of these :) Can you help to figure out how the import operation could have failed in this way? Is there a BZ for this already? Thank you, Laszlo ----- Original Message -----
From: "Jakub Bittner" <j.bittner@nbu.cz> To: users@ovirt.org Sent: Wednesday, August 28, 2013 5:12:23 PM Subject: Re: [Users] All VMs disappeared
Dne 27.8.2013 16:01, Laszlo Hornyak napsal(a):
Thank you Frank, I will investigate this issue.
----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 2:44:44 PM Subject: Re: [Users] All VMs disappeared
Hi Laszlo,
On 27.08.2013 14:24, Laszlo Hornyak wrote:
If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up. just to be sure, please note, that the task actually completed successfully from my point of view. The new VM (direved from the template clone task) is fully usable and on the webadmin the task is shown as "completed".
So maybe at some point in time the engine does get the job status wrong and the problem begins...
I need to look into this but I think it should be safe to remove this record while the engine is down. Make sure to make a backup if you try! OK, I did the following test:
- created a new VM from my custom template (with cloned Disk) - waited for the job to complete - manually deleted the related DB entry from the table "async_tasks" - set host/node to maintenance - rebootet engine
Result: The VM did NOT disappear! :-)
I think somehow ovirt-engine does get the results of the job wrong, thus it does not recognize that the job completed *successfully* and does a retry after reboot (which fails horribly and leads to disappearing VMs).
Thanks - Frank
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users Hi,
I manually deleted async_tasks and I dont have problem with disappearing of VMs. Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Frank, I was working on some priority tasks, but I am trying to save some time to identify the root cause of the problem this week. I do have some idea for the reason of the error but I did not sort out which one is right and which is wrong. Can you help me with - vdsm log - engine log from startup - the content of async_tasks table before engien startup (dump would be great) Thank you, Laszlo ----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Cc: "Laszlo Hornyak" <lhornyak@redhat.com> Sent: Tuesday, September 3, 2013 9:00:24 PM Subject: Re: [Users] All VMs disappeared
Hi Laszlo,
On 27.08.2013 16:01, Laszlo Hornyak wrote:
Thank you Frank, I will investigate this issue.
any update on this issue? It's 100% reproducible, even on a new installed RC2 host.
Thanks - Frank
I ran into the same issue when importing VMs into oVirt 3.2, assigning them permissions to a user such as admin would allow them to survive the reboot and not disappear. In 3.3 that does not seem to be the case. I opened a BZ but I think it got closed. Just for reference how'd you empty the async_tasks? Andrew. On Tue, Aug 27, 2013 at 10:44 PM, Frank Wall <fw@moov.de> wrote:
Hi Laszlo,
On 27.08.2013 14:24, Laszlo Hornyak wrote:
If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up.
just to be sure, please note, that the task actually completed successfully from my point of view. The new VM (direved from the template clone task) is fully usable and on the webadmin the task is shown as "completed".
So maybe at some point in time the engine does get the job status wrong and the problem begins...
I need to look into this but I think it should be safe to remove this
record while the engine is down. Make sure to make a backup if you try!
OK, I did the following test:
- created a new VM from my custom template (with cloned Disk) - waited for the job to complete - manually deleted the related DB entry from the table "async_tasks" - set host/node to maintenance - rebootet engine
Result: The VM did NOT disappear! :-)
I think somehow ovirt-engine does get the results of the job wrong, thus it does not recognize that the job completed *successfully* and does a retry after reboot (which fails horribly and leads to disappearing VMs).
Thanks - Frank
______________________________**_________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/**mailman/listinfo/users<http://lists.ovirt.org/mailman/listinfo/users>
Hi Andrew, On 06.09.2013 03:42, Andrew Lau wrote:
Just for reference how'd you empty the async_tasks?
on my oVirt engine I use the following steps to empty async_tasks: aio:~# su -l postgres -bash-4.2$ psql psql (9.2.4) postgres=# \c engine engine=# select task_id from async_tasks; task_id --------- [...should return one or more rows...] engine=# delete from async_tasks; Please make sure that all tasks completed before removing these rows. Regards - Frank
Hi Frank, Thanks, that seems to have done the trick. Andrew On Sat, Sep 7, 2013 at 3:10 AM, Frank Wall <fw@moov.de> wrote:
Hi Andrew,
On 06.09.2013 03:42, Andrew Lau wrote:
Just for reference how'd you empty the async_tasks?
on my oVirt engine I use the following steps to empty async_tasks:
aio:~# su -l postgres
-bash-4.2$ psql psql (9.2.4)
postgres=# \c engine
engine=# select task_id from async_tasks; task_id --------- [...should return one or more rows...]
engine=# delete from async_tasks;
Please make sure that all tasks completed before removing these rows.
Regards
- Frank ______________________________**_________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/**mailman/listinfo/users<http://lists.ovirt.org/mailman/listinfo/users>
On 27/08/13 15:44, Frank Wall wrote:
Hi Laszlo,
On 27.08.2013 14:24, Laszlo Hornyak wrote:
If I understand correctly, this is what happens to your installation, the AddVmFromTemplate command goes to fail and engine tries to compensate, and upon successful compensation you get VM records removed from DB and storage cleaned up. just to be sure, please note, that the task actually completed successfully from my point of view. The new VM (direved from the template clone task) is fully usable and on the webadmin the task is shown as "completed".
So maybe at some point in time the engine does get the job status wrong and the problem begins...
I need to look into this but I think it should be safe to remove this record while the engine is down. Make sure to make a backup if you try! OK, I did the following test:
- created a new VM from my custom template (with cloned Disk) - waited for the job to complete - manually deleted the related DB entry from the table "async_tasks" - set host/node to maintenance - rebootet engine
Result: The VM did NOT disappear! :-)
I think somehow ovirt-engine does get the results of the job wrong, thus it does not recognize that the job completed *successfully* and does a retry after reboot (which fails horribly and leads to disappearing VMs).
Thanks - Frank
Hi, I had the same experience with the latest beta version, ovirt-engine-3.3.0-1.el6.noarch After create VM from template was initiated, two rows where inserted in the table "async_tasks" . After the creation finished successfully (and the VM could be used), one row was deleted from the table but one was left there. If you restart ovirt-engine then the VMs (created from template) where deleted. Their disks were not deleted. The left over row was also deleted. If you manually delete the left-over row from the table before you restart ovirt-engine, then the VM is not deleted. Is there a BZ for this to monitor it's fix progress? regards, Giannis
----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 2:47:28 PM Subject: Re: [Users] All VMs disappeared
On 26.08.2013 23:40, Frank Wall wrote:
Every time I reboot ovirt-engine the VMs will disappear. No matter what I do before rebooting ovirt-engine - set storage to maintenance, set host to maintenance - it's always the same.
On IRC #ovirt I was asked to have a look at the "async_tasks" table while cloning a VM from a template. An entry is created in this table, but the vdsm_id is incorrect:
engine=# select task_id,action_type,status,vdsm_task_id from async_tasks; 6fa81c23-6f64-444c-bab0-f178fa9233d5 | 2 | 2 | 00000000-0000-0000-0000-000000000000
vdsm_task_id should not be "zero" for a long time. Once VDSM creates the relevant task, it should return engine the vdsm task id, and engine should update it in db. If you can compile/check from source - see http://gerrit.ovirt.org/#/c/17582/ - this should solve this issue. Yair
Maybe this empty vdsm_id is related to the disappearing vm issue?
Thanks - Frank _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 12:40:06 AM Subject: Re: [Users] All VMs disappeared
On 09.08.2013 09:48, Jakub Bittner wrote:
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
I have the same problem, but with a slightly different setup:
- fresh install of Fedora 19 (all updates installed) - fresh install of oVirt 3.3 RC (no upgrade) - All-in-One setup with only local storage - VMs installed from scratch (no import) - VM/node/engine uptime was about 22 hours before reboot
Now the problem is very the same as reported by Jakub: Every time I reboot ovirt-engine the VMs will disappear. No matter what I do before rebooting ovirt-engine - set storage to maintenance, set host to maintenance - it's always the same.
One interesting thing to note: Only VMs created from my custom template will disappear! If I create a VM from the "blank" template, it will NOT disappear.
Hi frank, I verified that commit c54353b11fb9ac2cb59cbc3e04b58e9114cbecba solved this issue as well.
Of course, the VM Disks can still be found in the filesystem, but it's a nightmare trying to guess what disk should be attached to what VM... I consider these VMs lost, since it's (fortunately) my test server, but still a day of work is lost.
Here are some interesting log entries from engine startup AFTER REBOOT:
2013-08-26 23:03:20,479 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-6-thread-7) Failing task with empty vdsm id AsyncTaskType copyImage : Task 7fa218c7-14c3-4a7f-97e8-341f18c6cb3d Parent Command AddVmFromTemplate
2013-08-26 23:03:20,497 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-6-thread-7) Calling updateTask for task with out vdsm id and AsyncTaskType copyImage : Task 7fa218c7-14c3-4a7f-97e8-341f18c6cb3d Parent Command AddVmFromTemplate Parameters class {3}
2013-08-26 23:03:20,506 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 4202990d-e85f-4935-a91b-a832bc8582cc
2013-08-26 23:03:20,787 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (pool-6-thread-7) [10025b02] Lock Acquired to object EngineLock [exclusiveLocks= key: vm1.example.com value: VM_NAME
2013-08-26 23:03:20,788 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-6-thread-7) [10025b02] CommandMultiAsyncTasks::AttachTask: Attaching task 00000000-0000-0000-0000-000000000000 to command 4202990d-e85f-4935-a91b-a832bc8582cc.
2013-08-26 23:03:20,796 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-6-thread-7) [10025b02] BaseAsyncTask::LogEndTaskFailure: Task 00000000-0000-0000-0000-000000000000 (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:
2013-08-26 23:03:20,805 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-6-thread-7) [10025b02] task with DB task ID 00000000-0000-0000-0000-000000000000 has empty vdsm task ID and is about to be cleared
2013-08-26 23:03:20,806 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) [10025b02] CommandAsyncTask::EndActionIfNecessary: All tasks of command 4202990d-e85f-4935-a91b-a832bc8582cc has ended -> executing EndAction
2013-08-26 23:03:20,807 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) [10025b02] CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 4202990d-e85f-4935-a91b-a832bc8582cc): calling EndAction .
Needless to say that I'm ABSOLUTELY SURE the template-copy tasks actually did succeed before rebooting, as I was using the VMs, installing software, rebooting, ... they were all running fine for 22 hours!
These messages exist for every missing VM. A show-stopper for 3.3 if you ask me :-/
Thanks - Frank _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 09/15/2013 10:19 PM, Yair Zaslavsky wrote:
----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: users@ovirt.org Sent: Tuesday, August 27, 2013 12:40:06 AM Subject: Re: [Users] All VMs disappeared
On 09.08.2013 09:48, Jakub Bittner wrote:
Iam running ovirt 3.3.0.beta1 on centos 6,4 and all our VMs disappeared from VMs tab. Nodes running on centos too. Repeating problem in log is:
I have the same problem, but with a slightly different setup:
- fresh install of Fedora 19 (all updates installed) - fresh install of oVirt 3.3 RC (no upgrade) - All-in-One setup with only local storage - VMs installed from scratch (no import) - VM/node/engine uptime was about 22 hours before reboot
Now the problem is very the same as reported by Jakub: Every time I reboot ovirt-engine the VMs will disappear. No matter what I do before rebooting ovirt-engine - set storage to maintenance, set host to maintenance - it's always the same.
One interesting thing to note: Only VMs created from my custom template will disappear! If I create a VM from the "blank" template, it will NOT disappear.
Hi frank, I verified that commit c54353b11fb9ac2cb59cbc3e04b58e9114cbecba solved this issue as well.
also worth mentioning this was fixed for 3.3 GA prior to its release.
Of course, the VM Disks can still be found in the filesystem, but it's a nightmare trying to guess what disk should be attached to what VM... I consider these VMs lost, since it's (fortunately) my test server, but still a day of work is lost.
Here are some interesting log entries from engine startup AFTER REBOOT:
2013-08-26 23:03:20,479 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-6-thread-7) Failing task with empty vdsm id AsyncTaskType copyImage : Task 7fa218c7-14c3-4a7f-97e8-341f18c6cb3d Parent Command AddVmFromTemplate
2013-08-26 23:03:20,497 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-6-thread-7) Calling updateTask for task with out vdsm id and AsyncTaskType copyImage : Task 7fa218c7-14c3-4a7f-97e8-341f18c6cb3d Parent Command AddVmFromTemplate Parameters class {3}
2013-08-26 23:03:20,506 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 4202990d-e85f-4935-a91b-a832bc8582cc
2013-08-26 23:03:20,787 INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (pool-6-thread-7) [10025b02] Lock Acquired to object EngineLock [exclusiveLocks= key: vm1.example.com value: VM_NAME
2013-08-26 23:03:20,788 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-6-thread-7) [10025b02] CommandMultiAsyncTasks::AttachTask: Attaching task 00000000-0000-0000-0000-000000000000 to command 4202990d-e85f-4935-a91b-a832bc8582cc.
2013-08-26 23:03:20,796 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-6-thread-7) [10025b02] BaseAsyncTask::LogEndTaskFailure: Task 00000000-0000-0000-0000-000000000000 (Parent Command AddVmFromTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:
2013-08-26 23:03:20,805 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-6-thread-7) [10025b02] task with DB task ID 00000000-0000-0000-0000-000000000000 has empty vdsm task ID and is about to be cleared
2013-08-26 23:03:20,806 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) [10025b02] CommandAsyncTask::EndActionIfNecessary: All tasks of command 4202990d-e85f-4935-a91b-a832bc8582cc has ended -> executing EndAction
2013-08-26 23:03:20,807 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-6-thread-7) [10025b02] CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 4202990d-e85f-4935-a91b-a832bc8582cc): calling EndAction .
Needless to say that I'm ABSOLUTELY SURE the template-copy tasks actually did succeed before rebooting, as I was using the VMs, installing software, rebooting, ... they were all running fine for 22 hours!
These messages exist for every missing VM. A show-stopper for 3.3 if you ask me :-/
Thanks - Frank _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi community. Centos 6.4 w/Ovirt 3.3.0-0.5.rc.el6. I have been having the same issue for a few months. (I thought it was just me.) For me, when I reboot, atleast ~50% of my VMs are gone. When I drill down to check the disks associated with the missing VMS them all show illegal states. I thought it might be hardware related so I replaced all my disks, and still have the same issue each time I reboot. It's gotten so bad that that before complete my patch/maint window by rebooting to test updates -- I have to export all the disks to an external LUN just to make sure, and then detach it from oVirt. I then re-import that LUN as an export share (via NFS) to get themn back into the system. Very very strange. Anyways, it till happens to me being fully upated as of 9/20/2013.
On 09/20/2013 07:29 PM, Michael McConachie wrote:
Hi community.
Centos 6.4 w/Ovirt 3.3.0-0.5.rc.el6.
I have been having the same issue for a few months. (I thought it was just me.) For me, when I reboot, atleast ~50% of my VMs are gone. When I drill down to check the disks associated with the missing VMS them all show illegal states.
I thought it might be hardware related so I replaced all my disks, and still have the same issue each time I reboot. It's gotten so bad that that before complete my patch/maint window by rebooting to test updates -- I have to export all the disks to an external LUN just to make sure, and then detach it from oVirt. I then re-import that LUN as an export share (via NFS) to get themn back into the system.
Very very strange. Anyways, it till happens to me being fully upated as of 9/20/2013.
this is a (sadly) known issue in 3.3 beta. it's supposed to be fixed for GA. please make sure you are using the correct repo. the GA version is 3.3.0-4.el6. reminder to upgrade by re-running engine-setup, not only yum update. thanks, Itamar
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Yair, On 15.09.2013 21:19, Yair Zaslavsky wrote:
I verified that commit c54353b11fb9ac2cb59cbc3e04b58e9114cbecba solved this issue as well.
I can confirm that the issue is solved. Thank you! ovirt-engine-3.3.0-4.fc19.noarch ovirt-engine-backend-3.3.0-4.fc19.noarch ovirt-engine-cli-3.3.0.4-1.20130829.git5ae0f2c.fc19.noarch ovirt-engine-dbscripts-3.3.0-4.fc19.noarch ovirt-engine-lib-3.3.0-4.fc19.noarch ovirt-engine-restapi-3.3.0-4.fc19.noarch ovirt-engine-sdk-python-3.3.0.6-1.fc19.noarch ovirt-engine-setup-3.3.0-4.fc19.noarch ovirt-engine-setup-plugin-allinone-3.3.0-4.fc19.noarch ovirt-engine-tools-3.3.0-4.fc19.noarch ovirt-engine-userportal-3.3.0-4.fc19.noarch ovirt-engine-webadmin-portal-3.3.0-4.fc19.noarch ovirt-host-deploy-1.1.1-1.fc19.noarch ovirt-host-deploy-java-1.1.1-1.fc19.noarch ovirt-host-deploy-offline-1.1.1-1.fc19.noarch ovirt-image-uploader-3.3.0-1.fc19.noarch ovirt-iso-uploader-3.3.0-1.fc19.noarch ovirt-log-collector-3.3.0-1.fc19.noarch Thanks - Frank
----- Original Message -----
From: "Frank Wall" <fw@moov.de> To: "Yair Zaslavsky" <yzaslavs@redhat.com> Cc: users@ovirt.org Sent: Sunday, September 22, 2013 10:32:42 PM Subject: Re: [Users] All VMs disappeared
Hi Yair,
On 15.09.2013 21:19, Yair Zaslavsky wrote:
I verified that commit c54353b11fb9ac2cb59cbc3e04b58e9114cbecba solved this issue as well.
I can confirm that the issue is solved. Thank you!
ovirt-engine-3.3.0-4.fc19.noarch ovirt-engine-backend-3.3.0-4.fc19.noarch ovirt-engine-cli-3.3.0.4-1.20130829.git5ae0f2c.fc19.noarch ovirt-engine-dbscripts-3.3.0-4.fc19.noarch ovirt-engine-lib-3.3.0-4.fc19.noarch ovirt-engine-restapi-3.3.0-4.fc19.noarch ovirt-engine-sdk-python-3.3.0.6-1.fc19.noarch ovirt-engine-setup-3.3.0-4.fc19.noarch ovirt-engine-setup-plugin-allinone-3.3.0-4.fc19.noarch ovirt-engine-tools-3.3.0-4.fc19.noarch ovirt-engine-userportal-3.3.0-4.fc19.noarch ovirt-engine-webadmin-portal-3.3.0-4.fc19.noarch ovirt-host-deploy-1.1.1-1.fc19.noarch ovirt-host-deploy-java-1.1.1-1.fc19.noarch ovirt-host-deploy-offline-1.1.1-1.fc19.noarch ovirt-image-uploader-3.3.0-1.fc19.noarch ovirt-iso-uploader-3.3.0-1.fc19.noarch ovirt-log-collector-3.3.0-1.fc19.noarch
Thanks - Frank
Thank you frank for your assistance on this issue and providing us clear reproducer. With you help we were able to track down this issue more efficiently. Cheers, Yair
participants (11)
-
Andrew Lau -
Frank Wall -
Greg Sheremeta -
Itamar Heim -
j.bittner＠nbu.cz -
Jakub Bittner
-
Kapetanakis Giannis -
Laszlo Hornyak -
Liron Aravot -
Michael McConachie -
Yair Zaslavsky