----- Original Message -----

From: "Neil" <nwilson123@gmail.com> To: users@ovirt.org Sent: Monday, October 15, 2012 9:27:24 AM Subject: [Users] Fwd: database issue after power failure

I've found out now that another technician accidentally deleted my psql engine db because he tried to install bacula on the same server. I have no backup of the DB because I was still in the process of setting them up and wasn't aware he was going to be installing onto this server.

I currently have 6 guests that are in a shutdown state because of a power failure over the weekend. It's using FC storage, only the db seems to be missing.

Is there anything I can do to recover from this disaster?

Hi Is this a new installation or one that was upgraded? If it is an upgrade IIRC we are backing up the database prior to the upgrade so maybe it is possible to search for this backup and getting an older version of your DB. If the DB is deleted and you have no backups you will probably have to reinstall. I recommend to backup your DB on a daily basis using our backup.sh script , you can use a simple cron job for that. Eli

Thanks.

Regards.

Neil.

---------- Forwarded message ---------- From: Neil <nwilson123@gmail.com> Date: Mon, Oct 15, 2012 at 9:10 AM Subject: database issue after power failure To: users@ovirt.org

Hi guys,

I've got what looks like a huge issue after a power failure.

In my server.log I'm seeing "Caused by: org.postgresql.util.PSQLException: FATAL: database "engine" does not exist" in my server.log and in my engine.log I'm seeing the following which also looks very bad.

"2012-10-15 09:08:41,357 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource"

Please could someone assist if possible, I'm currently logged in on #ovirt as well.

Thanks.

Regards.

Neil Wilson. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

I'm able to access the LUNS directly and I can access the guest LV filesystems, so I'm currently dumping the guest LV's to raw disk images, does anyone know how to import these again into ovirt? On Mon, Oct 15, 2012 at 11:49 AM, Neil <nwilson123@gmail.com> wrote:

On Mon, Oct 15, 2012 at 11:43 AM, Eli Mesika <emesika@redhat.com> wrote:

...

----- Original Message -----

...

From: "Neil" <nwilson123@gmail.com> To: users@ovirt.org Sent: Monday, October 15, 2012 9:27:24 AM Subject: [Users] Fwd: database issue after power failure

I've found out now that another technician accidentally deleted my psql engine db because he tried to install bacula on the same server. I have no backup of the DB because I was still in the process of setting them up and wasn't aware he was going to be installing onto this server.

I currently have 6 guests that are in a shutdown state because of a power failure over the weekend. It's using FC storage, only the db seems to be missing.

Is there anything I can do to recover from this disaster?

Hi Is this a new installation or one that was upgraded? If it is an upgrade IIRC we are backing up the database prior to the upgrade so maybe it is possible to search for this backup and getting an older version of your DB.

If the DB is deleted and you have no backups you will probably have to reinstall. I recommend to backup your DB on a daily basis using our backup.sh script , you can use a simple cron job for that.

Eli

...

Thanks.

Regards.

Neil.

---------- Forwarded message ---------- From: Neil <nwilson123@gmail.com> Date: Mon, Oct 15, 2012 at 9:10 AM Subject: database issue after power failure To: users@ovirt.org

Hi guys,

I've got what looks like a huge issue after a power failure.

In my server.log I'm seeing "Caused by: org.postgresql.util.PSQLException: FATAL: database "engine" does not exist" in my server.log and in my engine.log I'm seeing the following which also looks very bad.

"2012-10-15 09:08:41,357 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource"

Please could someone assist if possible, I'm currently logged in on #ovirt as well.

Thanks.

Regards.

Neil Wilson.

Hi Eli,

Thanks for coming back to me.

Unfortunately it wasn't an upgrade or installation, just raw db files were removed(rm -f /var/lib/pgsql/*) mid last week, and then there was power failure over the weekend and everything shut down and then refused to start.

I've tried running extundelete but it says the "space has been re-allocated" for the files under /var/lib/pgsql

The images should in theory still be fine because they are stored on a FC SAN, but I now need to somehow access the .ovf files, back them up, re-install the ovirt-engine from scratch re-initialise the storage and then import the .ovf files.

Anyone know how to access the .ovf files without having an engine? I can access the LUNS directly, but can't mount them because of the filesystem type being unknown.

Thanks.

Regards.

Neil.

On Mon, Oct 15, 2012 at 12:28 PM, Alexandre Santos <santosam72@gmail.com> wrote:

Look at the post about converting VMs from KVM to oVirt, I think it will help you.

Alex

2012/10/15 Neil <nwilson123@gmail.com>

...

I'm able to access the LUNS directly and I can access the guest LV filesystems, so I'm currently dumping the guest LV's to raw disk images, does anyone know how to import these again into ovirt?

On Mon, Oct 15, 2012 at 11:49 AM, Neil <nwilson123@gmail.com> wrote:

...

On Mon, Oct 15, 2012 at 11:43 AM, Eli Mesika <emesika@redhat.com> wrote:

...

----- Original Message -----

...

From: "Neil" <nwilson123@gmail.com> To: users@ovirt.org Sent: Monday, October 15, 2012 9:27:24 AM Subject: [Users] Fwd: database issue after power failure

I've found out now that another technician accidentally deleted my psql engine db because he tried to install bacula on the same server. I have no backup of the DB because I was still in the process of setting them up and wasn't aware he was going to be installing onto this server.

I currently have 6 guests that are in a shutdown state because of a power failure over the weekend. It's using FC storage, only the db seems to be missing.

Is there anything I can do to recover from this disaster?

Hi Is this a new installation or one that was upgraded? If it is an upgrade IIRC we are backing up the database prior to the upgrade so maybe it is possible to search for this backup and getting an older version of your DB.

If the DB is deleted and you have no backups you will probably have to reinstall. I recommend to backup your DB on a daily basis using our backup.sh script , you can use a simple cron job for that.

Eli

...

Thanks.

Regards.

Neil.

---------- Forwarded message ---------- From: Neil <nwilson123@gmail.com> Date: Mon, Oct 15, 2012 at 9:10 AM Subject: database issue after power failure To: users@ovirt.org

Hi guys,

I've got what looks like a huge issue after a power failure.

In my server.log I'm seeing "Caused by: org.postgresql.util.PSQLException: FATAL: database "engine" does not exist" in my server.log and in my engine.log I'm seeing the following which also looks very bad.

"2012-10-15 09:08:41,357 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-4) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource"

Please could someone assist if possible, I'm currently logged in on #ovirt as well.

Thanks.

Regards.

Neil Wilson.

Hi Eli,

Thanks for coming back to me.

Unfortunately it wasn't an upgrade or installation, just raw db files were removed(rm -f /var/lib/pgsql/*) mid last week, and then there was power failure over the weekend and everything shut down and then refused to start.

I've tried running extundelete but it says the "space has been re-allocated" for the files under /var/lib/pgsql

The images should in theory still be fine because they are stored on a FC SAN, but I now need to somehow access the .ovf files, back them up, re-install the ovirt-engine from scratch re-initialise the storage and then import the .ovf files.

Anyone know how to access the .ovf files without having an engine? I can access the LUNS directly, but can't mount them because of the filesystem type being unknown.

Thanks.

Regards.

Neil.

Thanks to everyone for assisting with this, we managed to dd the guest images to a NAS directly, re-install ovirt-engine from scratch as well as the hosts and re-create the used LUNS on the SAN, then create the new guests images the same size as they were previously and then dd the images back to the new guest images and we are all back up and running. For those who haven't got a DB backup of their live ovirt-engine, make one NOW because you don't want to have to do all of the above when a 1 minute backup could prevent it. Thanks! Regards. Neil Wilson.