On 02/06/2013 05:09 PM, Jim Kinney wrote:
bad admin: the pki restore did not include some vital parts (.keystore !)
SPM host is live now but others won't activate. Seems the database restore dropped a function:
Make sure you have the "postgresql-contrib" package installed, then try this: # psql -U postgres engine engine# create extension "uuid-ossp"; After that restart the engine.
2013-02-06 11:06:55,650 ERROR [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-3-thread-49) [58e7592f] Command org.ovirt.engine.core.bll.ActivateVdsCommand throw exception: org.springframework.jdbc.BadSqlGrammarException: CallableStatementCallback; bad SQL grammar [{call insert_entity_snapshot(?, ?, ?, ?, ?, ?, ?, ?)}]; nested exception is org.postgresql.util.PSQLException: ERROR: function uuid_generate_v1() does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts. Where: PL/pgSQL function "insert_entity_snapshot" line 4 at SQL statement at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:97) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:952) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:985) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:368) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:342) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:37) [engine-dal.jar:] at org.ovirt.engine.core.dao.BusinessEntitySnapshotDAODbFacadeImpl.save(BusinessEntitySnapshotDAODbFacadeImpl.java:84) [engine-dal.jar:] at org.ovirt.engine.core.bll.context.DefaultCompensationContext.stateChanged(DefaultCompensationContext.java:138) [engine-bll.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand$1.runInTransaction(ActivateVdsCommand.java:60) [engine-bll.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand$1.runInTransaction(ActivateVdsCommand.java:53) [engine-bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:] at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.ExecuteWithoutTransaction(CommandBase.java:804) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:896) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1203) [engine-bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:142) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:109) [engine-utils.jar:] at org.ovirt.engine.core.bll.CommandBase.Execute(CommandBase.java:911) [engine-bll.jar:] at org.ovirt.engine.core.bll.CommandBase.ExecuteAction(CommandBase.java:268) [engine-bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] Caused by: org.postgresql.util.PSQLException: ERROR: function uuid_generate_v1() does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts. Where: PL/pgSQL function "insert_entity_snapshot" line 4 at SQL statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062) [postgresql-jdbc.jar:] at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795) [postgresql-jdbc.jar:] at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) [postgresql-jdbc.jar:] at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) [postgresql-jdbc.jar:] at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367) [postgresql-jdbc.jar:] at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:360) [postgresql-jdbc.jar:] at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:297) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404) at org.springframework.jdbc.core.JdbcTemplate$5.doInCallableStatement(JdbcTemplate.java:987) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:936) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
On Wed, Feb 6, 2013 at 10:46 AM, Jim Kinney <jim.kinney@gmail.com <mailto:jim.kinney@gmail.com>> wrote:
On Wed, Feb 6, 2013 at 10:18 AM, Jim Kinney <jim.kinney@gmail.com <mailto:jim.kinney@gmail.com>> wrote:
progress. Restored pki files from backup. Still had to reset AdminPassword. Able to login to the gui. All hosts are "unresponsive". The SPM host is just totally locked (but fine from it's console - idle).
Tried to reinstall one of the hosts and got a new error message:
Error:
vmhost5:
* size must be between 0 and 50
vmhost2 is locked as SPM host and nothing seems to allow it to be reset. Found the following in engine.log:
2013-02-06 10:42:51,809 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( Vds: vmhost2 ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target 2013-02-06 10:42:51,818 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-57) Failed to decryptData must start with zero 2013-02-06 10:42:52,513 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptData must start with zero 2013-02-06 10:42:53,245 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-60) Failed to decryptData must start with zero
This repeats every few seconds.
None of the vmhosts have the usual vdsm running profiles. They all have:
19343 ? S< 0:00 /bin/bash -e /usr/share/vdsm/respawn --minlifetime 10 --daemon --masterpid /var/run/vdsm/respawn.pid /usr/share/vdsm/vdsm 19346 ? S<l 0:02 /usr/bin/python /usr/share/vdsm/vdsm 19366 ? S< 0:00 /usr/bin/sudo -n /usr/bin/python /usr/share/vdsm/supervdsmServer.py 414b69cd-383b-48af-a82a-7f1d042608e3 19346 19367 ? S<l 0:00 /usr/bin/python /usr/share/vdsm/supervdsmServer.py 414b69cd-383b-48af-a82a-7f1d042608e3 19346
Expected to see may [vdsmd] entries in ps ax output.
On Wed, Feb 6, 2013 at 9:42 AM, Juan Hernandez <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:
On 02/06/2013 03:32 PM, Jim Kinney wrote:
The pki folder is likely to be a problem but the backups folder is populated. Is there a way to remove client certs from hosts to restore access with a host add process?
If you don't have the pki folder you have two problems. First is that some data in the database is encrypted, namely the AdminPassword. That you can solve with "engine-config -s AdminPassword=interactive". Second is the certificates of the hosts, the easy way to solve that is to re-install them (from the ovirt-engine GUI, no need to re-install the operating system) that will generate new certificates.
On Feb 6, 2013 9:24 AM, "Juan Hernandez" <jhernand@redhat.com <mailto:jhernand@redhat.com> <mailto:jhernand@redhat.com <mailto:jhernand@redhat.com>>> wrote:
On 02/06/2013 03:02 PM, Jim Kinney wrote:
as things stand now:
I manually reinstalled 3.1, then dropped the engine database and restored from the backup. There were some errors at the end. Even though I used all the same passwords, the admin@internal account was not working. Used engine-config -s LocalAdminPassword='*****' to fix. On log in, everything is down, offline, unreachable. No hosts can be contacted. No storage is connected. Can't add a new host.
crud.
I copied the database backup and removed all the db creation part leaving just the data "copy into..." section (that was fun). Ran engine-cleanup then engine-setup then tried to restore just the data.
no joy there either.
The system is CentOS 6.3 as are the hosts. This ran wonderfully until I goofed trying to get the cli and sdk updated. Without the database working, I have no way to know what vm is what in the ISCSI LVM storage system to even export to another platform.
So I'm assuming my next step is panic (or total reinstall from bare iron?). I'm setting this up at work and today is my last day as I'm moving to a new job at a totally different organization. I'd hate to walk out and lose all the windows VMs and templates that were built over the last 2 months.
Do you still have the original backup of the database and the contents of the original /etc/pki/ovirt-engine directory? With those two things it is possible to recover.
I would suggest the following procedure:
1. Make a clean installation of 3.1, exactly the same version that you had before trying to update (make a backup of the database and of the /etc/pki/ovirt-engine directory before, just in case). During this installation use the answers that you used during the initial installation (specially the passwords).
2. Stop the engine, then drop and recover the database as you already did.
3. Restore the contents of the /etc/pki/ovirt-engine directory.
4. Start the engine.
You should be able to log in with the same credentials that you used in the original installation.
On Wed, Feb 6, 2013 at 8:43 AM, Jim Kinney <jim.kinney@gmail.com <mailto:jim.kinney@gmail.com> <mailto:jim.kinney@gmail.com <mailto:jim.kinney@gmail.com>> <mailto:jim.kinney@gmail.com <mailto:jim.kinney@gmail.com> <mailto:jim.kinney@gmail.com <mailto:jim.kinney@gmail.com>>>__> wrote:
added 3.2 lines to dre ovirt yum repo (and disabled 3.1 - probably not good) and did engine-upgrade.
Process choked at opening the CA cert and proceeded to "rollback". Didn't actually roll back as 3.1 repo was disabled.
System still has 3.2 installed. Did yum update to pull in the cli/sdk 3.2 (wish I had done that first!).
Engine starts but fails to open CA to run gui. found following in log:
2013-02-05 14:02:40,825 ERROR [org.ovirt.engine.core. engineencryptutils.____EncryptionUtils] (MSC service thread 1-16) Can't load keystore from file "/etc/pki/ovirt-engine/.____keystore".
IOException: DerInputStream.getLength(): lengthTag=109, too big. 2013-02-05 14:02:40,826 ERROR
[org.ovirt.engine.core.____engineencryptutils.____EncryptionUtils] (MSC
service thread 1-16) Failed to decrypt java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big. 2013-02-05 14:02:40,827 ERROR
[org.ovirt.engine.core.dal.____dbbroker.generic.____DBConfigUtils] (MSC
service thread 1-16) Failed to decrypt value for property TruststorePass will be used encrypted value 2013-02-05 14:02:40,829 WARN
[org.ovirt.engine.core.utils.____ConfigUtilsBase] (MSC
service thread 1-16) Could not find enum value for option: CertificatePassword 2013-02-05 14:02:40,830 ERROR
[org.ovirt.engine.core.____engineencryptutils.____EncryptionUtils] (MSC
service thread 1-16) Can't load keystore from file "/etc/pki/ovirt-engine/.____keystore". IOException:
DerInputStream.getLength(): lengthTag=109, too big. 2013-02-05 14:02:40,830 ERROR
[org.ovirt.engine.core.____engineencryptutils.____EncryptionUtils] (MSC
service thread 1-16) Failed to decrypt java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big. 2013-02-05 14:02:40,831 ERROR
[org.ovirt.engine.core.dal.____dbbroker.generic.____DBConfigUtils] (MSC
service thread 1-16) Failed to decrypt value for property LocalAdminPassword will be used encrypted value 2013-02-05 14:02:40,833 ERROR
[org.ovirt.engine.core.____engineencryptutils.____EncryptionUtils] (MSC
service thread 1-16) Can't load keystore from file "/etc/pki/ovirt-engine/.____keystore". IOException:
DerInputStream.getLength(): lengthTag=109, too big. 2013-02-05 14:02:40,834 ERROR
[org.ovirt.engine.core.____engineencryptutils.____EncryptionUtils] (MSC
service thread 1-16) Failed to decrypt java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
On Tue, Feb 5, 2013 at 6:11 AM, Michael Pasternak <mpastern@redhat.com <mailto:mpastern@redhat.com> <mailto:mpastern@redhat.com <mailto:mpastern@redhat.com>> <mailto:mpastern@redhat.com <mailto:mpastern@redhat.com> <mailto:mpastern@redhat.com <mailto:mpastern@redhat.com>>>> wrote:
Hi Jim,
On 02/04/2013 08:33 PM, Jim Kinney wrote: > I'm trying to setup a way to restart a large group of windows vms on a schedule. I'm getting a connection failure that seems related to the use of https but I'm not sure. > > error: __init__() got an unexpected keyword argument 'source_address'
This error is caused by running ovirt-sdk on a older version of python (less then python27), please upgrade your sdk/cli with one shipped in 3.2 (it's backward compatible to 3.1).
> > I ran: > ovirt-shell -A <path to server cert/certfile exported from browser> -c > > and my .ovirtshellrc is: > > [ovirt-shell] > username = "admin@internal" > url = https://my.internal.url/api > #insecure = False > #filter = False > #timeout = -1 > password = ********************** > > > I tried putting the ca_cert = <path to cert> but that clearly was not allowed in .ovirtshellrc
not related, but supported in 3.2 cli.
> > ideas? > -- > -- > James P. Kinney III > //// > ////Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten > the dog. > - Speech 11/23/1900 Mark Twain > //// > http://electjimkinney.org > http://heretothereideas.__blog__spot.com/ <http://blogspot.com/> <http://heretothereideas.__blogspot.com/ <http://heretothereideas.blogspot.com/>> > //// > > > > ___________________________________________________
> Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> <mailto:Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>> > http://lists.ovirt.org/____mailman/listinfo/users <http://lists.ovirt.org/__mailman/listinfo/users>
<http://lists.ovirt.org/__mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>>
--
Michael Pasternak RedHat, ENG-Virtualization R&D
-- -- James P. Kinney III //// ////Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain //// http://electjimkinney.org http://heretothereideas.__blog__spot.com/ <http://blogspot.com/>
<http://heretothereideas.__blogspot.com/ <http://heretothereideas.blogspot.com/>> ////
-- -- James P. Kinney III //// ////Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain //// http://electjimkinney.org http://heretothereideas.__blog__spot.com/ <http://blogspot.com/> <http://heretothereideas.__blogspot.com/ <http://heretothereideas.blogspot.com/>> ////
___________________________________________________
Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> http://lists.ovirt.org/____mailman/listinfo/users <http://lists.ovirt.org/__mailman/listinfo/users>
<http://lists.ovirt.org/__mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>>
-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
_________________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/__mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
-- -- James P. Kinney III //// ////Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain //// http://electjimkinney.org http://heretothereideas.blogspot.com/ ////
-- -- James P. Kinney III //// ////Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain //// http://electjimkinney.org http://heretothereideas.blogspot.com/ ////
-- -- James P. Kinney III //// ////Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain //// http://electjimkinney.org http://heretothereideas.blogspot.com/ ////
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta 3ºD, 28016 Madrid, Spain Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.