9:46 a.m.
After setting the .keystore owner and group owner to ovirt, and
rebooting, I now have a new error in engine.log
2013-04-08 02:39:16,787 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-95) Failed to decryptData must start with zero
2013-04-08 02:39:16,845 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-95) XML RPC error in command
GetCapabilitiesVDS ( Vds: transporter ), the error was:
java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException,
SunCertPathBuilderException: unable to find valid certification path
to requested target
Are there other files that may have been affected that I can also
correct ownership or permissions on?
On the host side, I get certificate unknown in vdsm.log
File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Thread-757809::ERROR::2013-04-08
02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
('172.16.23.8', 54489)
Traceback (most recent call last):
File "/usr/lib64/python2.7/SocketServer.py", line 582, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
line 66, in finish_request
request.do_handshake()
File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Is there a procedure for just re-establishing PKI and certs for the
engine and hosts?
On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl(a)redhat.com> wrote:
OK... you are running a very old version of engine (3.1).
The upgrade did not upgraded into 3.2, so nothing as far as I know should have been
changed.
But the .keystore permissions is owned by root now, so some other package (maybe
selinux-policy) changed permissions...
The simplest way to test is to:
# cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
# chown -R ovirt:ovirt /etc/pki/ovirt-engine
But if that file permissions was changed, I can only assume other files were also
changes...
Regards,
Alon
----- Original Message -----
> From: "Chris Smith" <whitehat237(a)gmail.com>
> To: "Alon Bar-Lev" <alonbl(a)redhat.com>
> Cc: Users(a)ovirt.org
> Sent: Sunday, April 7, 2013 11:51:17 AM
> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>
> I did a yum update and rebooted.
>
> engine-upgrade was run on 24-March
>
> When run now, it states that there are no updates available.
>
> [root@reliant ~]# engine-upgrade
> Loaded plugins: versionlock
> Checking for updates... (This may take several minutes)
> No updates available
>
>
> [root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log
> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
> pgpass file, fetching DB host value
> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
> pgpass file, fetching DB port value
> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
> pgpass file, fetching DB admin value
> 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates
> started
> 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started
> 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock
> completed successfully
> 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list
> of packages to upgrade
> 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-backend'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-backend-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-config'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-config-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-genericapi'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-genericapi-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-notification-service'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-notification-service-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-restapi'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-restapi-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-tools-common'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-tools-common-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-userportal'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-userportal-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
> command --> '/bin/rpm -q ovirt-engine-webadmin-portal'
> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
> ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
>
> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
> 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm
> -q ovirt-engine ovirt-engine-backend ovirt-engine-config
> ovirt-engine-genericapi ovirt-engine-notification-service
> ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal
> ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list
> 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output =
> 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr =
> 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0
> 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock
> completed successfully
> 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages
> marked for update
> 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages:
> 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root::
> ['ovirt-engine-3.1.0-4.fc17.noarch',
> 'ovirt-engine-backend-3.1.0-4.fc17.noarch',
> 'ovirt-engine-config-3.1.0-4.fc17.noarch',
> 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch',
> 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch',
> 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch',
> 'ovirt-engine-restapi-3.1.0-4.fc17.noarch',
> 'ovirt-engine-setup-3.1.0-4.fc17.noarch',
> 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch',
> 'ovirt-engine-userportal-3.1.0-4.fc17.noarch',
> 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch',
> 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch',
> 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch',
> 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch',
> 'vdsm-bootstrap-4.10.0-13.fc17.noarch']
> 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list
> updated completed successfully
> 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available
>
>
> Here's what's installed.
>
> [root@reliant yum.repos.d]# yum list installed | grep ovirt
> ovirt-engine.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-backend.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-cli.noarch 3.2.0.5-1.fc17
> @updates
> ovirt-engine-config.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-dbscripts.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-genericapi.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-notification-service.noarch
> 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-restapi.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-sdk.noarch 3.2.0.2-1.fc17
> @updates
> ovirt-engine-setup.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-tools-common.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-userportal.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17
> @ovirt-stable
> ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17
> @ovirt-stable
> ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17
> @ovirt-stable
> ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17
> @ovirt-stable
> ovirt-release-fedora.noarch 4-2
> @/ovirt-release-fedora.noarch
>
> On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl(a)redhat.com> wrote:
> > How exactly did you upgrade?
> >
> > Usually yum upgrade will not touch ovirt-engine packages as it is in yum
> > version lock.
> > From which version to which version have you upgraded?
> > Have you run engine-upgrade utility?
> > If you did not, please run it.
> > If you did, please attach logs from
> > /var/log/ovirt-engine/ovirt-engine-upgrade*
> >
> > Thanks!
> >
> > ----- Original Message -----
> >> From: "Chris Smith" <whitehat237(a)gmail.com>
> >> To: Users(a)ovirt.org
> >> Sent: Sunday, April 7, 2013 5:09:46 AM
> >> Subject: [Users] Certificates and PKI seem to be broken after yum update
> >>
> >> I have lost the ability to manage the hosts or VM's using ovirt
> >> engine web interface after performing yum update on the ovirt-engine
> >> host, and on one Fedora 17 host. The data center is offline, and I
> >> can't place the hosts into maintenance mode. I don't think that
there
> >> are any actions I can perform in the web interface at all.
> >>
> >> From the logs it seems that PKI is broken between the engine and the
> >> hosts.
> >>
> >> I am wondering how I can restore or re-generate all of the
> >> certificates and get the hosts communicating with the ovirt-engine
> >> again so that I can bring the data center back online.
> >>
> >> I found this page which deals with changing the engine hostname, and
> >> thus re-creating the certificates and keystore on the ovirt-engine
> >> node, and was wondering if this could help. Could I follow this
> >> process but keep the same hostname for the ovirt-engine node?
> >>
> >> http://wiki.ovirt.org/How_to_change_engine_host_name
> >>
> >> Currently I have 3 VM's running on two hosts. The VM's are up, but
I
> >> can't do anything with them in ovirt-engine.
> >>
> >>
> >> Here's the latest activity from engine.log from the ovirt-engine node:
> >>
> >> 2013-04-06 21:58:47,472 ERROR
> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
> >> (QuartzScheduler_Worker-61) Failed to
> >> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
> >> (Permission denied)
> >> 2013-04-06 21:58:47,478 ERROR
> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
> >> (QuartzScheduler_Worker-62) Can't load keystore from file
> >> "/etc/pki/ovirt-engine/.keystore".:
java.io.FileNotFoundException:
> >> /etc/pki/ovirt-engine/.keystore (Permission denied)
> >> at java.io.FileInputStream.open(Native Method)
> >> [rt.jar:1.7.0_09-icedtea]
> >> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> >> [rt.jar:1.7.0_09-icedtea]
> >> at
> >>
org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214)
> >> [engine-encryptutils.jar:]
> >> at
> >>
org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139)
> >> [engine-encryptutils.jar:]
> >> at
> >>
org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169)
> >> [engine-dal.jar:]
> >> at
> >>
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121)
> >> [engine-dal.jar:]
> >> at
> >>
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164)
> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
> >> at
> >>
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31)
> >> [engine-dal.jar:]
> >> at
> >>
org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219)
> >> [engine-vdsbroker.jar:]
> >> at
> >>
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168)
> >> [engine-utils.jar:]
> >> at
> >>
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107)
> >> [engine-utils.jar:]
> >> at
> >>
org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215)
> >> [engine-vdsbroker.jar:]
> >> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown
> >> Source) [:1.7.0_09-icedtea]
> >> at
> >>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> [rt.jar:1.7.0_09-icedtea]
> >> at java.lang.reflect.Method.invoke(Method.java:601)
> >> [rt.jar:1.7.0_09-icedtea]
> >> at
> >>
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64)
> >> [engine-scheduler.jar:]
> >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> >> [quartz.jar:]
> >> at
> >>
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> >> [quartz.jar:]
> >>
> >> 2013-04-06 21:58:47,576 ERROR
> >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
> >> (QuartzScheduler_Worker-61) XML RPC error in command
> >> GetCapabilitiesVDS ( Vds: defiant ), the error was:
> >> java.util.concurrent.ExecutionException:
> >> java.lang.reflect.InvocationTargetException,
> >> SSLPeerUnverifiedException: peer not authenticated
> >> 2013-04-06 21:58:47,606 ERROR
> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
> >> (QuartzScheduler_Worker-62) Failed to
> >> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
> >> (Permission denied)
> >> 2013-04-06 21:58:47,671 ERROR
> >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
> >> (QuartzScheduler_Worker-62) XML RPC error in command
> >> GetCapabilitiesVDS ( Vds: transporter ), the error was:
> >> java.util.concurrent.ExecutionException:
> >> java.lang.reflect.InvocationTargetException,
> >> SSLPeerUnverifiedException: peer not authenticated
> >>
> >>
> >> Here's the message I seem to get over and over on the fedora 17 host in
> >> vdsm.log
> >>
> >> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
> >> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
> >> Thread-562520::ERROR::2013-04-06
> >> 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client
> >> ('172.16.23.8', 36127)
> >> Traceback (most recent call last):
> >> File "/usr/lib64/python2.7/SocketServer.py", line 582, in
> >> process_request_thread
> >> self.finish_request(request, client_address)
> >> File
"/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
> >> line 66, in finish_request
> >> request.do_handshake()
> >> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
> >> self._sslobj.do_handshake()
> >>
> >> I'm also wondering about the permission denied on the .keystore
> >> directory. What should the permissions be? Here's what they are
> >> currently.
> >>
> >> [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore
> >> -rwxr-x---. root root unconfined_u:object_r:cert_t:s0
> >> /etc/pki/ovirt-engine/.keystore
> >>
> >> I also seem to have a backup of the ovirt-engine directory at the time
> >> the update was performed, but replacing ovirt-engine with the backup
> >> does no good.
> >>
> >> I appreciate any assistance, and please let me know what other
> >> information I can post to help with this.
> >>
> >> Thanks
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/users
> >>
>