[Users] Certificates and PKI seem to be broken after yum update

I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all. I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online. I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node? http://wiki.ovirt.org/How_to_change_engine_host_name Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine. Here's the latest activity from engine.log from the ovirt-engine node: 2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] 2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated Here's the message I seem to get over and over on the fedora 17 host in vdsm.log SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently. [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good. I appreciate any assistance, and please let me know what other information I can post to help with this. Thanks