[Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online. I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node? http://wiki.ovirt.org/How_to_change_engine_host_name Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine. Here's the latest activity from engine.log from the ovirt-engine node: 2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] 2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated Here's the message I seem to get over and over on the fedora 17 host in vdsm.log SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently. [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good. I appreciate any assistance, and please let me know what other information I can post to help with this. Thanks
How exactly did you upgrade? Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock.
From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks! ----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: Users@ovirt.org Sent: Sunday, April 7, 2013 5:09:46 AM Subject: [Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online.
I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node?
http://wiki.ovirt.org/How_to_change_engine_host_name
Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine.
Here's the latest activity from engine.log from the ovirt-engine node:
2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated
Here's the message I seem to get over and over on the fedora 17 host in vdsm.log
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake()
I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently.
[root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore
I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good.
I appreciate any assistance, and please let me know what other information I can post to help with this.
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I did a yum update and rebooted. engine-upgrade was run on 24-March When run now, it states that there are no updates available. [root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available [root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available Here's what's installed. [root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
How exactly did you upgrade?
Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock. From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks!
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: Users@ovirt.org Sent: Sunday, April 7, 2013 5:09:46 AM Subject: [Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online.
I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node?
http://wiki.ovirt.org/How_to_change_engine_host_name
Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine.
Here's the latest activity from engine.log from the ovirt-engine node:
2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated
Here's the message I seem to get over and over on the fedora 17 host in vdsm.log
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake()
I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently.
[root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore
I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good.
I appreciate any assistance, and please let me know what other information I can post to help with this.
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
OK... you are running a very old version of engine (3.1). The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed. But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions... The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine But if that file permissions was changed, I can only assume other files were also changes... Regards, Alon ----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Sunday, April 7, 2013 11:51:17 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
I did a yum update and rebooted.
engine-upgrade was run on 24-March
When run now, it states that there are no updates available.
[root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available
[root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
Here's what's installed.
[root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
How exactly did you upgrade?
Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock. From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks!
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: Users@ovirt.org Sent: Sunday, April 7, 2013 5:09:46 AM Subject: [Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online.
I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node?
http://wiki.ovirt.org/How_to_change_engine_host_name
Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine.
Here's the latest activity from engine.log from the ovirt-engine node:
2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated
Here's the message I seem to get over and over on the fedora 17 host in vdsm.log
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake()
I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently.
[root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore
I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good.
I appreciate any assistance, and please let me know what other information I can post to help with this.
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
After setting the .keystore owner and group owner to ovirt, and rebooting, I now have a new error in engine.log 2013-04-08 02:39:16,787 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decryptData must start with zero 2013-04-08 02:39:16,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target Are there other files that may have been affected that I can also correct ownership or permissions on? On the host side, I get certificate unknown in vdsm.log File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-757809::ERROR::2013-04-08 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 54489) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Is there a procedure for just re-establishing PKI and certs for the engine and hosts? On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
OK... you are running a very old version of engine (3.1).
The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed.
But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions...
The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine
But if that file permissions was changed, I can only assume other files were also changes...
Regards, Alon
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Sunday, April 7, 2013 11:51:17 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
I did a yum update and rebooted.
engine-upgrade was run on 24-March
When run now, it states that there are no updates available.
[root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available
[root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
Here's what's installed.
[root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
How exactly did you upgrade?
Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock. From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks!
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: Users@ovirt.org Sent: Sunday, April 7, 2013 5:09:46 AM Subject: [Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online.
I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node?
http://wiki.ovirt.org/How_to_change_engine_host_name
Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine.
Here's the latest activity from engine.log from the ovirt-engine node:
2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated
Here's the message I seem to get over and over on the fedora 17 host in vdsm.log
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake()
I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently.
[root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore
I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good.
I appreciate any assistance, and please let me know what other information I can post to help with this.
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or does not contain the /etc/pki/ovirt-engine/ca.pem certificate. Unfortunately, the pki administration is weak in current implementation, you can trace the installation script and checkout the calls to installCA.sh to how to reproduce, please note that password are encrypted in database using the private key locate in .keystore so if you are to re-generate anything remember to keep the engine private key. However, if you succeed in login, the remaining problem you have is the .truststore permissions and/or content. Regards, Alon Bar-Lev. ----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Monday, April 8, 2013 9:46:46 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
After setting the .keystore owner and group owner to ovirt, and rebooting, I now have a new error in engine.log
2013-04-08 02:39:16,787 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decryptData must start with zero 2013-04-08 02:39:16,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target
Are there other files that may have been affected that I can also correct ownership or permissions on?
On the host side, I get certificate unknown in vdsm.log
File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-757809::ERROR::2013-04-08 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 54489) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Is there a procedure for just re-establishing PKI and certs for the engine and hosts?
On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
OK... you are running a very old version of engine (3.1).
The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed.
But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions...
The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine
But if that file permissions was changed, I can only assume other files were also changes...
Regards, Alon
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Sunday, April 7, 2013 11:51:17 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
I did a yum update and rebooted.
engine-upgrade was run on 24-March
When run now, it states that there are no updates available.
[root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available
[root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
Here's what's installed.
[root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
How exactly did you upgrade?
Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock. From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks!
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: Users@ovirt.org Sent: Sunday, April 7, 2013 5:09:46 AM Subject: [Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online.
I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node?
http://wiki.ovirt.org/How_to_change_engine_host_name
Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine.
Here's the latest activity from engine.log from the ovirt-engine node:
2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated
Here's the message I seem to get over and over on the fedora 17 host in vdsm.log
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake()
I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently.
[root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore
I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good.
I appreciate any assistance, and please let me know what other information I can post to help with this.
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I checked the .truststore on the ovirt engine, and it seems fine. [root@reliant ovirt-engine]# ls -l .truststore -rwxr-x---. 1 ovirt ovirt 918 Apr 6 21:56 .truststore It's not zero bytes anyway. It's also the same size as the .truststore in the ovirt engine backups. [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} \; -rwxr-x---. 1 ovirt ovirt 918 Aug 26 2012 ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore -rwxr-x---. 1 root root 918 Mar 24 12:42 ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore I haven't looked at the installCA.sh script yet. On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
Unfortunately, the pki administration is weak in current implementation, you can trace the installation script and checkout the calls to installCA.sh to how to reproduce, please note that password are encrypted in database using the private key locate in .keystore so if you are to re-generate anything remember to keep the engine private key.
However, if you succeed in login, the remaining problem you have is the .truststore permissions and/or content.
Regards, Alon Bar-Lev.
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Monday, April 8, 2013 9:46:46 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
After setting the .keystore owner and group owner to ovirt, and rebooting, I now have a new error in engine.log
2013-04-08 02:39:16,787 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decryptData must start with zero 2013-04-08 02:39:16,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target
Are there other files that may have been affected that I can also correct ownership or permissions on?
On the host side, I get certificate unknown in vdsm.log
File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-757809::ERROR::2013-04-08 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 54489) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Is there a procedure for just re-establishing PKI and certs for the engine and hosts?
On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
OK... you are running a very old version of engine (3.1).
The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed.
But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions...
The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine
But if that file permissions was changed, I can only assume other files were also changes...
Regards, Alon
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Sunday, April 7, 2013 11:51:17 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
I did a yum update and rebooted.
engine-upgrade was run on 24-March
When run now, it states that there are no updates available.
[root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available
[root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
Here's what's installed.
[root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
How exactly did you upgrade?
Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock. From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks!
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: Users@ovirt.org Sent: Sunday, April 7, 2013 5:09:46 AM Subject: [Users] Certificates and PKI seem to be broken after yum update
I have lost the ability to manage the hosts or VM's using ovirt engine web interface after performing yum update on the ovirt-engine host, and on one Fedora 17 host. The data center is offline, and I can't place the hosts into maintenance mode. I don't think that there are any actions I can perform in the web interface at all.
From the logs it seems that PKI is broken between the engine and the hosts.
I am wondering how I can restore or re-generate all of the certificates and get the hosts communicating with the ovirt-engine again so that I can bring the data center back online.
I found this page which deals with changing the engine hostname, and thus re-creating the certificates and keystore on the ovirt-engine node, and was wondering if this could help. Could I follow this process but keep the same hostname for the ovirt-engine node?
http://wiki.ovirt.org/How_to_change_engine_host_name
Currently I have 3 VM's running on two hosts. The VM's are up, but I can't do anything with them in ovirt-engine.
Here's the latest activity from engine.log from the ovirt-engine node:
2013-04-06 21:58:47,472 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,478 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Can't load keystore from file "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea] at java.io.FileInputStream.<init>(FileInputStream.java:138) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) [engine-encryptutils.jar:] at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) [engine-encryptutils.jar:] at org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) [engine-dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) [engine-dal.jar:] at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) [engine-dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) [engine-vdsbroker.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) [engine-vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) [engine-scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
2013-04-06 21:58:47,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( Vds: defiant ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated 2013-04-06 21:58:47,606 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore (Permission denied) 2013-04-06 21:58:47,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLPeerUnverifiedException: peer not authenticated
Here's the message I seem to get over and over on the fedora 17 host in vdsm.log
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-562520::ERROR::2013-04-06 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 36127) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake()
I'm also wondering about the permission denied on the .keystore directory. What should the permissions be? Here's what they are currently.
[root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 /etc/pki/ovirt-engine/.keystore
I also seem to have a backup of the ovirt-engine directory at the time the update was performed, but replacing ovirt-engine with the backup does no good.
I appreciate any assistance, and please let me know what other information I can post to help with this.
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
If it would be easier than re-setting up the certificates, I'm also willing to just start over and rebuild, but I would like to export the VM's I have first. One of them is a spacewalk server, another runs DNS, and DHCP for my test network, and I have an asterisk server. I would like to avoid having to re-create all of them. The VM's are up and running now, so I could export all of the configurations / backup the file systems, etc. Preferably I could export the VM's to an NFS export domain, or a mounted NFS share so that I can import them to the new storage domain, after I run engine-cleanup and get everything set back up. Is there an easy way to do this? Is it possible to create and attach an NFS export domain directly from the CLI without access to the ovirt manager without communication between the manager and hosts due to the pki issue? Can I export the VM's directly from the hosts to a standard NFS share? Is there an equivalent xml and image file for the VM? My storage domain is iscsi and is served out from another server over 4 bonded 1 Gbps copper links. On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith <whitehat237@gmail.com> wrote:
I checked the .truststore on the ovirt engine, and it seems fine.
[root@reliant ovirt-engine]# ls -l .truststore -rwxr-x---. 1 ovirt ovirt 918 Apr 6 21:56 .truststore
It's not zero bytes anyway.
It's also the same size as the .truststore in the ovirt engine backups.
[root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} \; -rwxr-x---. 1 ovirt ovirt 918 Aug 26 2012 ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore -rwxr-x---. 1 root root 918 Mar 24 12:42 ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
I haven't looked at the installCA.sh script yet.
On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
Unfortunately, the pki administration is weak in current implementation, you can trace the installation script and checkout the calls to installCA.sh to how to reproduce, please note that password are encrypted in database using the private key locate in .keystore so if you are to re-generate anything remember to keep the engine private key.
However, if you succeed in login, the remaining problem you have is the .truststore permissions and/or content.
Regards, Alon Bar-Lev.
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Monday, April 8, 2013 9:46:46 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
After setting the .keystore owner and group owner to ovirt, and rebooting, I now have a new error in engine.log
2013-04-08 02:39:16,787 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decryptData must start with zero 2013-04-08 02:39:16,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target
Are there other files that may have been affected that I can also correct ownership or permissions on?
On the host side, I get certificate unknown in vdsm.log
File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-757809::ERROR::2013-04-08 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 54489) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Is there a procedure for just re-establishing PKI and certs for the engine and hosts?
On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
OK... you are running a very old version of engine (3.1).
The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed.
But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions...
The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine
But if that file permissions was changed, I can only assume other files were also changes...
Regards, Alon
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Sunday, April 7, 2013 11:51:17 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
I did a yum update and rebooted.
engine-upgrade was run on 24-March
When run now, it states that there are no updates available.
[root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available
[root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
Here's what's installed.
[root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
How exactly did you upgrade?
Usually yum upgrade will not touch ovirt-engine packages as it is in yum version lock. From which version to which version have you upgraded? Have you run engine-upgrade utility? If you did not, please run it. If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*
Thanks!
----- Original Message ----- > From: "Chris Smith" <whitehat237@gmail.com> > To: Users@ovirt.org > Sent: Sunday, April 7, 2013 5:09:46 AM > Subject: [Users] Certificates and PKI seem to be broken after yum > update > > I have lost the ability to manage the hosts or VM's using ovirt > engine web interface after performing yum update on the ovirt-engine > host, and on one Fedora 17 host. The data center is offline, and I > can't place the hosts into maintenance mode. I don't think that there > are any actions I can perform in the web interface at all. > > From the logs it seems that PKI is broken between the engine and the > hosts. > > I am wondering how I can restore or re-generate all of the > certificates and get the hosts communicating with the ovirt-engine > again so that I can bring the data center back online. > > I found this page which deals with changing the engine hostname, and > thus re-creating the certificates and keystore on the ovirt-engine > node, and was wondering if this could help. Could I follow this > process but keep the same hostname for the ovirt-engine node? > > http://wiki.ovirt.org/How_to_change_engine_host_name > > Currently I have 3 VM's running on two hosts. The VM's are up, but I > can't do anything with them in ovirt-engine. > > > Here's the latest activity from engine.log from the ovirt-engine node: > > 2013-04-06 21:58:47,472 ERROR > [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] > (QuartzScheduler_Worker-61) Failed to > decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore > (Permission denied) > 2013-04-06 21:58:47,478 ERROR > [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] > (QuartzScheduler_Worker-62) Can't load keystore from file > "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: > /etc/pki/ovirt-engine/.keystore (Permission denied) > at java.io.FileInputStream.open(Native Method) > [rt.jar:1.7.0_09-icedtea] > at java.io.FileInputStream.<init>(FileInputStream.java:138) > [rt.jar:1.7.0_09-icedtea] > at > org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) > [engine-encryptutils.jar:] > at > org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) > [engine-encryptutils.jar:] > at > org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) > [engine-dal.jar:] > at > org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) > [engine-dal.jar:] > at > org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) > [engine-dal.jar:] > at > org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) > [engine-dal.jar:] > at > org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) > [engine-dal.jar:] > at > org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) > [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > at > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) > [engine-dal.jar:] > at > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) > [engine-dal.jar:] > at > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) > [engine-dal.jar:] > at > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) > [engine-dal.jar:] > at > org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) > [engine-dal.jar:] > at > org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) > [engine-dal.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) > [engine-vdsbroker.jar:] > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) > [engine-utils.jar:] > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) > [engine-utils.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) > [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown > Source) [:1.7.0_09-icedtea] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) > [rt.jar:1.7.0_09-icedtea] > at > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) > [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > [quartz.jar:] > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > [quartz.jar:] > > 2013-04-06 21:58:47,576 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] > (QuartzScheduler_Worker-61) XML RPC error in command > GetCapabilitiesVDS ( Vds: defiant ), the error was: > java.util.concurrent.ExecutionException: > java.lang.reflect.InvocationTargetException, > SSLPeerUnverifiedException: peer not authenticated > 2013-04-06 21:58:47,606 ERROR > [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] > (QuartzScheduler_Worker-62) Failed to > decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore > (Permission denied) > 2013-04-06 21:58:47,671 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] > (QuartzScheduler_Worker-62) XML RPC error in command > GetCapabilitiesVDS ( Vds: transporter ), the error was: > java.util.concurrent.ExecutionException: > java.lang.reflect.InvocationTargetException, > SSLPeerUnverifiedException: peer not authenticated > > > Here's the message I seem to get over and over on the fedora 17 host in > vdsm.log > > SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL > routines:SSL3_READ_BYTES:sslv3 alert certificate unknown > Thread-562520::ERROR::2013-04-06 > 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client > ('172.16.23.8', 36127) > Traceback (most recent call last): > File "/usr/lib64/python2.7/SocketServer.py", line 582, in > process_request_thread > self.finish_request(request, client_address) > File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", > line 66, in finish_request > request.do_handshake() > File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake > self._sslobj.do_handshake() > > I'm also wondering about the permission denied on the .keystore > directory. What should the permissions be? Here's what they are > currently. > > [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore > -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 > /etc/pki/ovirt-engine/.keystore > > I also seem to have a backup of the ovirt-engine directory at the time > the update was performed, but replacing ovirt-engine with the backup > does no good. > > I appreciate any assistance, and please let me know what other > information I can post to help with this. > > Thanks > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
You should ask these question in separate thread so people may pick them up. For the .truststore, try to remove it and then execute: # rm -f /etc/pki/ovirt-engine/.truststore # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass -file /etc/pki/ovirt-engine/certs/ca.der -keystore /etc/pki/ovirt-engine/.truststore -storepass mypass # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore It should recreate the truststore with the ca certificate you have. ----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Thursday, April 18, 2013 7:18:27 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
If it would be easier than re-setting up the certificates, I'm also willing to just start over and rebuild, but I would like to export the VM's I have first. One of them is a spacewalk server, another runs DNS, and DHCP for my test network, and I have an asterisk server. I would like to avoid having to re-create all of them.
The VM's are up and running now, so I could export all of the configurations / backup the file systems, etc.
Preferably I could export the VM's to an NFS export domain, or a mounted NFS share so that I can import them to the new storage domain, after I run engine-cleanup and get everything set back up. Is there an easy way to do this? Is it possible to create and attach an NFS export domain directly from the CLI without access to the ovirt manager without communication between the manager and hosts due to the pki issue? Can I export the VM's directly from the hosts to a standard NFS share?
Is there an equivalent xml and image file for the VM?
My storage domain is iscsi and is served out from another server over 4 bonded 1 Gbps copper links.
On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith <whitehat237@gmail.com> wrote:
I checked the .truststore on the ovirt engine, and it seems fine.
[root@reliant ovirt-engine]# ls -l .truststore -rwxr-x---. 1 ovirt ovirt 918 Apr 6 21:56 .truststore
It's not zero bytes anyway.
It's also the same size as the .truststore in the ovirt engine backups.
[root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} \; -rwxr-x---. 1 ovirt ovirt 918 Aug 26 2012 ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore -rwxr-x---. 1 root root 918 Mar 24 12:42 ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
I haven't looked at the installCA.sh script yet.
On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
Unfortunately, the pki administration is weak in current implementation, you can trace the installation script and checkout the calls to installCA.sh to how to reproduce, please note that password are encrypted in database using the private key locate in .keystore so if you are to re-generate anything remember to keep the engine private key.
However, if you succeed in login, the remaining problem you have is the .truststore permissions and/or content.
Regards, Alon Bar-Lev.
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Monday, April 8, 2013 9:46:46 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
After setting the .keystore owner and group owner to ovirt, and rebooting, I now have a new error in engine.log
2013-04-08 02:39:16,787 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decryptData must start with zero 2013-04-08 02:39:16,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( Vds: transporter ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target
Are there other files that may have been affected that I can also correct ownership or permissions on?
On the host side, I get certificate unknown in vdsm.log
File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown Thread-757809::ERROR::2013-04-08 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client ('172.16.23.8', 54489) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Is there a procedure for just re-establishing PKI and certs for the engine and hosts?
On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl@redhat.com> wrote:
OK... you are running a very old version of engine (3.1).
The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed.
But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions...
The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine
But if that file permissions was changed, I can only assume other files were also changes...
Regards, Alon
----- Original Message -----
From: "Chris Smith" <whitehat237@gmail.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: Users@ovirt.org Sent: Sunday, April 7, 2013 11:51:17 AM Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
I did a yum update and rebooted.
engine-upgrade was run on 24-March
When run now, it states that there are no updates available.
[root@reliant ~]# engine-upgrade Loaded plugins: versionlock Checking for updates... (This may take several minutes) No updates available
[root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB host value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB port value 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing pgpass file, fetching DB admin value 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates started 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock completed successfully 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list of packages to upgrade 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-backend' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-backend-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-config' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-config-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-genericapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-genericapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-notification-service' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-notification-service-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-restapi' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-restapi-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-tools-common' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-tools-common-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-userportal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-userportal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing command --> '/bin/rpm -q ovirt-engine-webadmin-portal' 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm -q ovirt-engine ovirt-engine-backend ovirt-engine-config ovirt-engine-genericapi ovirt-engine-notification-service ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages marked for update 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: ['ovirt-engine-3.1.0-4.fc17.noarch', 'ovirt-engine-backend-3.1.0-4.fc17.noarch', 'ovirt-engine-config-3.1.0-4.fc17.noarch', 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', 'ovirt-engine-setup-3.1.0-4.fc17.noarch', 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list updated completed successfully 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
Here's what's installed.
[root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification-service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch
On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl@redhat.com> wrote: > How exactly did you upgrade? > > Usually yum upgrade will not touch ovirt-engine packages as it is in > yum > version lock. > From which version to which version have you upgraded? > Have you run engine-upgrade utility? > If you did not, please run it. > If you did, please attach logs from > /var/log/ovirt-engine/ovirt-engine-upgrade* > > Thanks! > > ----- Original Message ----- >> From: "Chris Smith" <whitehat237@gmail.com> >> To: Users@ovirt.org >> Sent: Sunday, April 7, 2013 5:09:46 AM >> Subject: [Users] Certificates and PKI seem to be broken after yum >> update >> >> I have lost the ability to manage the hosts or VM's using ovirt >> engine web interface after performing yum update on the >> ovirt-engine >> host, and on one Fedora 17 host. The data center is offline, and I >> can't place the hosts into maintenance mode. I don't think that >> there >> are any actions I can perform in the web interface at all. >> >> From the logs it seems that PKI is broken between the engine and >> the >> hosts. >> >> I am wondering how I can restore or re-generate all of the >> certificates and get the hosts communicating with the ovirt-engine >> again so that I can bring the data center back online. >> >> I found this page which deals with changing the engine hostname, >> and >> thus re-creating the certificates and keystore on the ovirt-engine >> node, and was wondering if this could help. Could I follow this >> process but keep the same hostname for the ovirt-engine node? >> >> http://wiki.ovirt.org/How_to_change_engine_host_name >> >> Currently I have 3 VM's running on two hosts. The VM's are up, but >> I >> can't do anything with them in ovirt-engine. >> >> >> Here's the latest activity from engine.log from the ovirt-engine >> node: >> >> 2013-04-06 21:58:47,472 ERROR >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> (QuartzScheduler_Worker-61) Failed to >> decryptjava.io.FileNotFoundException: >> /etc/pki/ovirt-engine/.keystore >> (Permission denied) >> 2013-04-06 21:58:47,478 ERROR >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> (QuartzScheduler_Worker-62) Can't load keystore from file >> "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: >> /etc/pki/ovirt-engine/.keystore (Permission denied) >> at java.io.FileInputStream.open(Native Method) >> [rt.jar:1.7.0_09-icedtea] >> at java.io.FileInputStream.<init>(FileInputStream.java:138) >> [rt.jar:1.7.0_09-icedtea] >> at >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) >> [engine-encryptutils.jar:] >> at >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) >> [engine-encryptutils.jar:] >> at >> org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) >> [engine-dal.jar:] >> at >> org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) >> [engine-dal.jar:] >> at >> org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> at >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) >> [engine-dal.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) >> [engine-vdsbroker.jar:] >> at >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) >> [engine-utils.jar:] >> at >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) >> [engine-utils.jar:] >> at >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) >> [engine-vdsbroker.jar:] >> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown >> Source) [:1.7.0_09-icedtea] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> [rt.jar:1.7.0_09-icedtea] >> at java.lang.reflect.Method.invoke(Method.java:601) >> [rt.jar:1.7.0_09-icedtea] >> at >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) >> [engine-scheduler.jar:] >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> [quartz.jar:] >> at >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> [quartz.jar:] >> >> 2013-04-06 21:58:47,576 ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] >> (QuartzScheduler_Worker-61) XML RPC error in command >> GetCapabilitiesVDS ( Vds: defiant ), the error was: >> java.util.concurrent.ExecutionException: >> java.lang.reflect.InvocationTargetException, >> SSLPeerUnverifiedException: peer not authenticated >> 2013-04-06 21:58:47,606 ERROR >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> (QuartzScheduler_Worker-62) Failed to >> decryptjava.io.FileNotFoundException: >> /etc/pki/ovirt-engine/.keystore >> (Permission denied) >> 2013-04-06 21:58:47,671 ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] >> (QuartzScheduler_Worker-62) XML RPC error in command >> GetCapabilitiesVDS ( Vds: transporter ), the error was: >> java.util.concurrent.ExecutionException: >> java.lang.reflect.InvocationTargetException, >> SSLPeerUnverifiedException: peer not authenticated >> >> >> Here's the message I seem to get over and over on the fedora 17 >> host in >> vdsm.log >> >> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL >> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown >> Thread-562520::ERROR::2013-04-06 >> 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client >> ('172.16.23.8', 36127) >> Traceback (most recent call last): >> File "/usr/lib64/python2.7/SocketServer.py", line 582, in >> process_request_thread >> self.finish_request(request, client_address) >> File >> "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", >> line 66, in finish_request >> request.do_handshake() >> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake >> self._sslobj.do_handshake() >> >> I'm also wondering about the permission denied on the .keystore >> directory. What should the permissions be? Here's what they are >> currently. >> >> [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore >> -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 >> /etc/pki/ovirt-engine/.keystore >> >> I also seem to have a backup of the ovirt-engine directory at the >> time >> the update was performed, but replacing ovirt-engine with the >> backup >> does no good. >> >> I appreciate any assistance, and please let me know what other >> information I can post to help with this. >> >> Thanks >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >>
participants (2)
-
Alon Bar-Lev -
Chris Smith