Postgresql failure (was: Ovirt Engine Issue)
On Mon, Dec 7, 2015 at 4:00 PM, Melissa Mesler <melissa@aasteel.com> wrote:
Here is what I retrieved from the postresql logs:
LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 1637) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 2632) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure
Also, when I did the sync on the ovirt vm it was powered off. I know not to do live syncs as there could be issues. That's why I'm so confused that there's a pr oblem.
No idea, adding Eli and changing subject. Best,
On 12/7/15 1:03 AM, Yedidyah Bar David wrote:
On Mon, Dec 7, 2015 at 12:09 AM, Melissa Mesler <melissa@aasteel.com> wrote:
Hopefully someone can give me a good idea of how to troubleshoot an ovirt issue. I am running 3.5 on CentOS 6.7. Here is the issue I'm having. I have Ovirt running as a virtual machine in KVM. I recently moved (with virtsync) the vm to rebuild the host. Once the host was rebuilt I moved ovirt back. Now ovirt-engine starts but is inaccessible. In the logs it says this:
2015-12-06 15:57:24,558 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource
I did notice that I can't get postgresql to start.
What happens when you try? Check/post also pg logs please, from /var/lib/pgsql/data/pg_log/ .
Everything worked perfectly fine before. Any ideas?
Was the copy done from a live machine? From a snapshot? Perhaps it wasn't consistent? Need to check pg logs.
Best,
-- Didi
----- Original Message -----
From: "Yedidyah Bar David" <didi@redhat.com> To: "Melissa Mesler" <melissa@aasteel.com>, "Eli Mesika" <emesika@redhat.com> Cc: users@ovirt.org Sent: Monday, December 7, 2015 4:14:49 PM Subject: Postgresql failure (was: [ovirt-users] Ovirt Engine Issue)
On Mon, Dec 7, 2015 at 4:00 PM, Melissa Mesler <melissa@aasteel.com> wrote:
Here is what I retrieved from the postresql logs:
LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 1637) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 2632) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure
Also, when I did the sync on the ovirt vm it was powered off. I know not to do live syncs as there could be issues. That's why I'm so confused that there's a pr oblem.
No idea, adding Eli and changing subject.
Seems as database is corrupted This is a serious issue that someone with DBA knowledge should handle http://blog.endpoint.com/2014/11/when-postgres-will-not-start.html Remember to do a physical copy of your database file to a different partition or server before you start Eli
Best,
On 12/7/15 1:03 AM, Yedidyah Bar David wrote:
On Mon, Dec 7, 2015 at 12:09 AM, Melissa Mesler <melissa@aasteel.com> wrote:
Hopefully someone can give me a good idea of how to troubleshoot an ovirt issue. I am running 3.5 on CentOS 6.7. Here is the issue I'm having. I have Ovirt running as a virtual machine in KVM. I recently moved (with virtsync) the vm to rebuild the host. Once the host was rebuilt I moved ovirt back. Now ovirt-engine starts but is inaccessible. In the logs it says this:
2015-12-06 15:57:24,558 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource
I did notice that I can't get postgresql to start.
What happens when you try? Check/post also pg logs please, from /var/lib/pgsql/data/pg_log/ .
Everything worked perfectly fine before. Any ideas?
Was the copy done from a live machine? From a snapshot? Perhaps it wasn't consistent? Need to check pg logs.
Best,
-- Didi
This is a multi-part message in MIME format. --------------090202020108020901020009 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit I might end up just rebuilding my ovirt which isn't a huge deal as my set up is pretty simple. I do have a question as to why this would happen. Can anyone give me guidance there? I would understand if I forcefully shut down the system but I didn't. I always shut my systems down clean so the database portion should've shutdown that way also. Just trying to understand more of why possibly this would happen so that I could avoid this in the future. --- Melissa Mesler On 12/7/15 8:14 AM, Yedidyah Bar David wrote:
On Mon, Dec 7, 2015 at 4:00 PM, Melissa Mesler <melissa@aasteel.com> wrote:
Here is what I retrieved from the postresql logs:
LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 1637) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 2632) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure
Also, when I did the sync on the ovirt vm it was powered off. I know not to do live syncs as there could be issues. That's why I'm so confused that there's a pr oblem. No idea, adding Eli and changing subject.
Best,
On 12/7/15 1:03 AM, Yedidyah Bar David wrote:
On Mon, Dec 7, 2015 at 12:09 AM, Melissa Mesler <melissa@aasteel.com> wrote:
Hopefully someone can give me a good idea of how to troubleshoot an ovirt issue. I am running 3.5 on CentOS 6.7. Here is the issue I'm having. I have Ovirt running as a virtual machine in KVM. I recently moved (with virtsync) the vm to rebuild the host. Once the host was rebuilt I moved ovirt back. Now ovirt-engine starts but is inaccessible. In the logs it says this:
2015-12-06 15:57:24,558 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource
I did notice that I can't get postgresql to start.
What happens when you try? Check/post also pg logs please, from /var/lib/pgsql/data/pg_log/ .
Everything worked perfectly fine before. Any ideas?
Was the copy done from a live machine? From a snapshot? Perhaps it wasn't consistent? Need to check pg logs.
Best,
--------------090202020108020901020009 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <small>I might end up just rebuilding my ovirt which isn't a huge deal as my set up is pretty simple.<br> <br> I do have a question as to why this would happen. Can anyone give me guidance there? I would understand if I forcefully shut down the system but I didn't. I always shut my systems down clean so the database portion should've shutdown that way also. Just trying to understand more of why possibly this would happen so that I could avoid this in the future.<br> <br> </small> <pre class="moz-signature" cols="72">--- Melissa Mesler </pre> <div class="moz-cite-prefix">On 12/7/15 8:14 AM, Yedidyah Bar David wrote:<br> </div> <blockquote cite="mid:CAHRwYXtrUK1XVMRKTnHGA40ML0w--1XxArm4CKBZAfmvq2X0zQ@mail.gmail.com" type="cite"> <pre wrap="">On Mon, Dec 7, 2015 at 4:00 PM, Melissa Mesler <a class="moz-txt-link-rfc2396E" href="mailto:melissa@aasteel.com"><melissa@aasteel.com></a> wrote: </pre> <blockquote type="cite"> <pre wrap="">Here is what I retrieved from the postresql logs: LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 1637) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure LOG: database system was interrupted; last known up at 2015-12-04 10:57:43 CST LOG: unexpected pageaddr 0/69564000 in log file 0, segment 107, offset 5652480 LOG: invalid primary checkpoint record LOG: unexpected pageaddr 0/6952A000 in log file 0, segment 107, offset 5414912 LOG: invalid secondary checkpoint record PANIC: could not locate a valid checkpoint record LOG: startup process (PID 2632) was terminated by signal 6: Aborted LOG: aborting startup due to startup process failure Also, when I did the sync on the ovirt vm it was powered off. I know not to do live syncs as there could be issues. That's why I'm so confused that there's a pr oblem. </pre> </blockquote> <pre wrap=""> No idea, adding Eli and changing subject. Best, </pre> <blockquote type="cite"> <pre wrap=""> On 12/7/15 1:03 AM, Yedidyah Bar David wrote: On Mon, Dec 7, 2015 at 12:09 AM, Melissa Mesler <a class="moz-txt-link-rfc2396E" href="mailto:melissa@aasteel.com"><melissa@aasteel.com></a> wrote: Hopefully someone can give me a good idea of how to troubleshoot an ovirt issue. I am running 3.5 on CentOS 6.7. Here is the issue I'm having. I have Ovirt running as a virtual machine in KVM. I recently moved (with virtsync) the vm to rebuild the host. Once the host was rebuilt I moved ovirt back. Now ovirt-engine starts but is inaccessible. In the logs it says this: 2015-12-06 15:57:24,558 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource I did notice that I can't get postgresql to start. What happens when you try? Check/post also pg logs please, from /var/lib/pgsql/data/pg_log/ . Everything worked perfectly fine before. Any ideas? Was the copy done from a live machine? From a snapshot? Perhaps it wasn't consistent? Need to check pg logs. Best, </pre> </blockquote> <pre wrap=""> </pre> </blockquote> <br> </body> </html> --------------090202020108020901020009--
participants (3)
-
Eli Mesika -
Melissa Mesler -
Yedidyah Bar David