[Users] Fwd: database issue after power failure

[Neil]

On Mon, Oct 15, 2012 at 12:28 PM, Alexandre Santos <santosam72 at gmail.com> wrote:
> Look at the post about converting VMs from KVM to oVirt, I think it will
> help you.
>
> Alex
>
> 2012/10/15 Neil <nwilson123 at gmail.com>
>>
>> I'm able to access the LUNS directly and I can access the guest LV
>> filesystems, so I'm currently dumping the guest LV's to raw disk
>> images, does anyone know how to import these again into ovirt?
>>
>> On Mon, Oct 15, 2012 at 11:49 AM, Neil <nwilson123 at gmail.com> wrote:
>> > On Mon, Oct 15, 2012 at 11:43 AM, Eli Mesika <emesika at redhat.com> wrote:
>> >>
>> >>
>> >> ----- Original Message -----
>> >>> From: "Neil" <nwilson123 at gmail.com>
>> >>> To: users at ovirt.org
>> >>> Sent: Monday, October 15, 2012 9:27:24 AM
>> >>> Subject: [Users] Fwd: database issue after power failure
>> >>>
>> >>> I've found out now that another technician accidentally deleted my
>> >>> psql engine db because he tried to install bacula on the same server.
>> >>> I have no backup of the DB because I was still in the process of
>> >>> setting them up and wasn't aware he was going to be installing onto
>> >>> this server.
>> >>>
>> >>> I currently have 6 guests that are in a shutdown state because of a
>> >>> power failure over the weekend. It's using FC storage, only the db
>> >>> seems to be missing.
>> >>>
>> >>> Is there anything I can do to recover from this disaster?
>> >>
>> >> Hi
>> >> Is this a new installation or one that was upgraded?
>> >> If it is an upgrade IIRC we are backing up the database prior to the
>> >> upgrade so maybe it is possible to search for this backup and getting an
>> >> older version of your DB.
>> >>
>> >> If the DB is deleted and you have no backups you will probably have to
>> >> reinstall.
>> >> I recommend to backup your DB on a daily basis using our backup.sh
>> >> script , you can use a simple cron job for that.
>> >>
>> >> Eli
>> >>
>> >>
>> >>>
>> >>> Thanks.
>> >>>
>> >>> Regards.
>> >>>
>> >>> Neil.
>> >>>
>> >>>
>> >>> ---------- Forwarded message ----------
>> >>> From: Neil <nwilson123 at gmail.com>
>> >>> Date: Mon, Oct 15, 2012 at 9:10 AM
>> >>> Subject: database issue after power failure
>> >>> To: users at ovirt.org
>> >>>
>> >>>
>> >>> Hi guys,
>> >>>
>> >>> I've got what looks like a huge issue after a power failure.
>> >>>
>> >>> In my server.log I'm seeing "Caused by:
>> >>> org.postgresql.util.PSQLException: FATAL: database "engine" does not
>> >>> exist" in my server.log and in my engine.log I'm seeing the following
>> >>> which also looks very bad.
>> >>>
>> >>> "2012-10-15 09:08:41,357 ERROR [org.ovirt.engine.core.bll.Backend]
>> >>> (MSC service thread 1-4) Error in getting DB connection. The database
>> >>> is inaccessible. Original exception is:
>> >>> DataAccessResourceFailureException: Error retreiving database
>> >>> metadata; nested exception is
>> >>> org.springframework.jdbc.support.MetaDataAccessException: Could not
>> >>> get Connection for extracting meta data; nested exception is
>> >>> org.springframework.jdbc.CannotGetJdbcConnectionException: Could not
>> >>> get JDBC Connection; nested exception is java.sql.SQLException:
>> >>> javax.resource.ResourceException: IJ000453: Unable to get managed
>> >>> connection for java:/ENGINEDataSource"
>> >>>
>> >>> Please could someone assist if possible, I'm currently logged in on
>> >>> #ovirt as well.
>> >>>
>> >>> Thanks.
>> >>>
>> >>> Regards.
>> >>>
>> >>> Neil Wilson.
>> >
>> > Hi Eli,
>> >
>> > Thanks for coming back to me.
>> >
>> > Unfortunately it wasn't an upgrade or installation, just raw db files
>> > were removed(rm -f /var/lib/pgsql/*) mid last week, and then there was
>> > power failure over the weekend and everything shut down and then
>> > refused to start.
>> >
>> > I've tried running extundelete but it says the "space has been
>> > re-allocated" for the files under /var/lib/pgsql
>> >
>> > The images should in theory still be fine because they are stored on a
>> > FC SAN, but I now need to somehow access the .ovf files, back them up,
>> > re-install the ovirt-engine from scratch re-initialise the storage and
>> > then import the .ovf files.
>> >
>> > Anyone know how to access the .ovf files without having an engine? I
>> > can access the LUNS directly, but can't mount them because of the
>> > filesystem type being unknown.
>> >
>> > Thanks.
>> >
>> > Regards.
>> >
>> > Neil.

Thanks to everyone for assisting with this, we managed to dd the guest
images to a NAS directly, re-install ovirt-engine from scratch as well
as the hosts and re-create the used LUNS on the SAN, then create the
new guests images the same size as they were previously and then dd
the images back to the new guest images and we are all back up and
running.

For those who haven't got a DB backup of their live ovirt-engine, make
one NOW because you don't want to have to do all of the above when a 1
minute backup could prevent it.

Thanks!

Regards.

Neil Wilson.