[Users] Certificates and PKI seem to be broken after yum update
Chris Smith
whitehat237 at gmail.com
Thu Apr 18 22:45:23 UTC 2013
On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith <whitehat237 at gmail.com> wrote:
> I made a backup of the .truststore, and then followed the steps and
> then rebooted both the ovirt-engine and one of the hosts, and
> everything worked properly.
>
> If I run it again, or enter the wrong password it throws an error
> about the key store already existing, or that the password was wrong
> so I'm pretty sure it's good.
>
> vdsm.log on the host still shows:
>
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/SocketServer.py", line 582, in
> process_request_thread
> self.finish_request(request, client_address)
> File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
> line 66, in finish_request
> request.do_handshake()
> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
> self._sslobj.do_handshake()
> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>
> engine.log on the host shows:
>
> 2013-04-18 18:42:43,632 ERROR
> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
> (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
> 2013-04-18 18:42:43,642 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
> (QuartzScheduler_Worker-68) XML RPC error in command
> GetCapabilitiesVDS ( Vds: transporter ), the error was:
> java.util.concurrent.ExecutionException:
> java.lang.reflect.InvocationTargetException,
> SunCertPathBuilderException: unable to find valid certification path
> to requested target
>
>
> On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev <alonbl at redhat.com> wrote:
>>
>> You should ask these question in separate thread so people may pick them up.
>>
>> For the .truststore, try to remove it and then execute:
>>
>> # rm -f /etc/pki/ovirt-engine/.truststore
>> # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass -file /etc/pki/ovirt-engine/certs/ca.der -keystore /etc/pki/ovirt-engine/.truststore -storepass mypass
>> # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
>>
>> It should recreate the truststore with the ca certificate you have.
>>
>> ----- Original Message -----
>>> From: "Chris Smith" <whitehat237 at gmail.com>
>>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>> Cc: Users at ovirt.org
>>> Sent: Thursday, April 18, 2013 7:18:27 AM
>>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>>
>>> If it would be easier than re-setting up the certificates, I'm also
>>> willing to just start over and rebuild, but I would like to export the
>>> VM's I have first.
>>> One of them is a spacewalk server, another runs DNS, and DHCP for my
>>> test network, and I have an asterisk server. I would like to avoid
>>> having to re-create all of them.
>>>
>>> The VM's are up and running now, so I could export all of the
>>> configurations / backup the file systems, etc.
>>>
>>> Preferably I could export the VM's to an NFS export domain, or a
>>> mounted NFS share so that I can import them to the new storage domain,
>>> after I run engine-cleanup and get everything set back up. Is there
>>> an easy way to do this? Is it possible to create and attach an NFS
>>> export domain directly from the CLI without access to the ovirt
>>> manager without communication between the manager and hosts due to the
>>> pki issue? Can I export the VM's directly from the hosts to a
>>> standard NFS share?
>>>
>>> Is there an equivalent xml and image file for the VM?
>>>
>>> My storage domain is iscsi and is served out from another server over
>>> 4 bonded 1 Gbps copper links.
>>>
>>>
>>>
>>> On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith <whitehat237 at gmail.com> wrote:
>>> > I checked the .truststore on the ovirt engine, and it seems fine.
>>> >
>>> > [root at reliant ovirt-engine]# ls -l .truststore
>>> > -rwxr-x---. 1 ovirt ovirt 918 Apr 6 21:56 .truststore
>>> >
>>> > It's not zero bytes anyway.
>>> >
>>> > It's also the same size as the .truststore in the ovirt engine backups.
>>> >
>>> > [root at reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l
>>> > {} \;
>>> > -rwxr-x---. 1 ovirt ovirt 918 Aug 26 2012
>>> > ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
>>> > -rwxr-x---. 1 root root 918 Mar 24 12:42
>>> > ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
>>> >
>>> > I haven't looked at the installCA.sh script yet.
>>> >
>>> > On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev <alonbl at redhat.com> wrote:
>>> >> This error means that the /etc/pki/ovirt-engine/.truststore is unreadable
>>> >> or does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
>>> >>
>>> >> Unfortunately, the pki administration is weak in current implementation,
>>> >> you can trace the installation script and checkout the calls to
>>> >> installCA.sh to how to reproduce, please note that password are encrypted
>>> >> in database using the private key locate in .keystore so if you are to
>>> >> re-generate anything remember to keep the engine private key.
>>> >>
>>> >> However, if you succeed in login, the remaining problem you have is the
>>> >> .truststore permissions and/or content.
>>> >>
>>> >> Regards,
>>> >> Alon Bar-Lev.
>>> >>
>>> >> ----- Original Message -----
>>> >>> From: "Chris Smith" <whitehat237 at gmail.com>
>>> >>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>> >>> Cc: Users at ovirt.org
>>> >>> Sent: Monday, April 8, 2013 9:46:46 AM
>>> >>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum
>>> >>> update
>>> >>>
>>> >>> After setting the .keystore owner and group owner to ovirt, and
>>> >>> rebooting, I now have a new error in engine.log
>>> >>>
>>> >>> 2013-04-08 02:39:16,787 ERROR
>>> >>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>> >>> (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
>>> >>> 2013-04-08 02:39:16,845 ERROR
>>> >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>> >>> (QuartzScheduler_Worker-95) XML RPC error in command
>>> >>> GetCapabilitiesVDS ( Vds: transporter ), the error was:
>>> >>> java.util.concurrent.ExecutionException:
>>> >>> java.lang.reflect.InvocationTargetException,
>>> >>> SunCertPathBuilderException: unable to find valid certification path
>>> >>> to requested target
>>> >>>
>>> >>> Are there other files that may have been affected that I can also
>>> >>> correct ownership or permissions on?
>>> >>>
>>> >>> On the host side, I get certificate unknown in vdsm.log
>>> >>>
>>> >>> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>>> >>> self._sslobj.do_handshake()
>>> >>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>>> >>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>> >>> Thread-757809::ERROR::2013-04-08
>>> >>> 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
>>> >>> ('172.16.23.8', 54489)
>>> >>> Traceback (most recent call last):
>>> >>> File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>>> >>> process_request_thread
>>> >>> self.finish_request(request, client_address)
>>> >>> File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>>> >>> line 66, in finish_request
>>> >>> request.do_handshake()
>>> >>> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>>> >>> self._sslobj.do_handshake()
>>> >>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>>> >>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>> >>>
>>> >>> Is there a procedure for just re-establishing PKI and certs for the
>>> >>> engine and hosts?
>>> >>>
>>> >>> On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl at redhat.com> wrote:
>>> >>> >
>>> >>> > OK... you are running a very old version of engine (3.1).
>>> >>> >
>>> >>> > The upgrade did not upgraded into 3.2, so nothing as far as I know
>>> >>> > should
>>> >>> > have been changed.
>>> >>> >
>>> >>> > But the .keystore permissions is owned by root now, so some other
>>> >>> > package
>>> >>> > (maybe selinux-policy) changed permissions...
>>> >>> >
>>> >>> > The simplest way to test is to:
>>> >>> > # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
>>> >>> > # chown -R ovirt:ovirt /etc/pki/ovirt-engine
>>> >>> >
>>> >>> > But if that file permissions was changed, I can only assume other files
>>> >>> > were also changes...
>>> >>> >
>>> >>> > Regards,
>>> >>> > Alon
>>> >>> >
>>> >>> > ----- Original Message -----
>>> >>> >> From: "Chris Smith" <whitehat237 at gmail.com>
>>> >>> >> To: "Alon Bar-Lev" <alonbl at redhat.com>
>>> >>> >> Cc: Users at ovirt.org
>>> >>> >> Sent: Sunday, April 7, 2013 11:51:17 AM
>>> >>> >> Subject: Re: [Users] Certificates and PKI seem to be broken after yum
>>> >>> >> update
>>> >>> >>
>>> >>> >> I did a yum update and rebooted.
>>> >>> >>
>>> >>> >> engine-upgrade was run on 24-March
>>> >>> >>
>>> >>> >> When run now, it states that there are no updates available.
>>> >>> >>
>>> >>> >> [root at reliant ~]# engine-upgrade
>>> >>> >> Loaded plugins: versionlock
>>> >>> >> Checking for updates... (This may take several minutes)
>>> >>> >> No updates available
>>> >>> >>
>>> >>> >>
>>> >>> >> [root at reliant ovirt-engine]# cat
>>> >>> >> ovirt-engine-upgrade_2013_03_24_12_04_06.log
>>> >>> >> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
>>> >>> >> pgpass file, fetching DB host value
>>> >>> >> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
>>> >>> >> pgpass file, fetching DB port value
>>> >>> >> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
>>> >>> >> pgpass file, fetching DB admin value
>>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list
>>> >>> >> updates
>>> >>> >> started
>>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock
>>> >>> >> started
>>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock
>>> >>> >> completed successfully
>>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list
>>> >>> >> of packages to upgrade
>>> >>> >> 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock
>>> >>> >> started
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-backend'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-backend-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-config'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-config-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-genericapi'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-genericapi-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-notification-service'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-notification-service-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-restapi'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-restapi-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-tools-common'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-tools-common-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-userportal'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-userportal-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>>> >>> >> command --> '/bin/rpm -q ovirt-engine-webadmin-portal'
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>>> >>> >> ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
>>> >>> >>
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm
>>> >>> >> -q ovirt-engine ovirt-engine-backend ovirt-engine-config
>>> >>> >> ovirt-engine-genericapi ovirt-engine-notification-service
>>> >>> >> ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal
>>> >>> >> ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list
>>> >>> >> 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output =
>>> >>> >> 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr =
>>> >>> >> 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0
>>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock
>>> >>> >> completed successfully
>>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages
>>> >>> >> marked for update
>>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed
>>> >>> >> packages:
>>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root::
>>> >>> >> ['ovirt-engine-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-backend-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-config-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-restapi-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-setup-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-userportal-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch',
>>> >>> >> 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch',
>>> >>> >> 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch',
>>> >>> >> 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch',
>>> >>> >> 'vdsm-bootstrap-4.10.0-13.fc17.noarch']
>>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list
>>> >>> >> updated completed successfully
>>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates
>>> >>> >> available
>>> >>> >>
>>> >>> >>
>>> >>> >> Here's what's installed.
>>> >>> >>
>>> >>> >> [root at reliant yum.repos.d]# yum list installed | grep ovirt
>>> >>> >> ovirt-engine.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-backend.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-cli.noarch 3.2.0.5-1.fc17
>>> >>> >> @updates
>>> >>> >> ovirt-engine-config.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-dbscripts.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-genericapi.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-notification-service.noarch
>>> >>> >> 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-restapi.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-sdk.noarch 3.2.0.2-1.fc17
>>> >>> >> @updates
>>> >>> >> ovirt-engine-setup.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-tools-common.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-userportal.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17
>>> >>> >> @ovirt-stable
>>> >>> >> ovirt-release-fedora.noarch 4-2
>>> >>> >> @/ovirt-release-fedora.noarch
>>> >>> >>
>>> >>> >> On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alonbl at redhat.com>
>>> >>> >> wrote:
>>> >>> >> > How exactly did you upgrade?
>>> >>> >> >
>>> >>> >> > Usually yum upgrade will not touch ovirt-engine packages as it is in
>>> >>> >> > yum
>>> >>> >> > version lock.
>>> >>> >> > From which version to which version have you upgraded?
>>> >>> >> > Have you run engine-upgrade utility?
>>> >>> >> > If you did not, please run it.
>>> >>> >> > If you did, please attach logs from
>>> >>> >> > /var/log/ovirt-engine/ovirt-engine-upgrade*
>>> >>> >> >
>>> >>> >> > Thanks!
>>> >>> >> >
>>> >>> >> > ----- Original Message -----
>>> >>> >> >> From: "Chris Smith" <whitehat237 at gmail.com>
>>> >>> >> >> To: Users at ovirt.org
>>> >>> >> >> Sent: Sunday, April 7, 2013 5:09:46 AM
>>> >>> >> >> Subject: [Users] Certificates and PKI seem to be broken after yum
>>> >>> >> >> update
>>> >>> >> >>
>>> >>> >> >> I have lost the ability to manage the hosts or VM's using ovirt
>>> >>> >> >> engine web interface after performing yum update on the
>>> >>> >> >> ovirt-engine
>>> >>> >> >> host, and on one Fedora 17 host. The data center is offline, and I
>>> >>> >> >> can't place the hosts into maintenance mode. I don't think that
>>> >>> >> >> there
>>> >>> >> >> are any actions I can perform in the web interface at all.
>>> >>> >> >>
>>> >>> >> >> From the logs it seems that PKI is broken between the engine and
>>> >>> >> >> the
>>> >>> >> >> hosts.
>>> >>> >> >>
>>> >>> >> >> I am wondering how I can restore or re-generate all of the
>>> >>> >> >> certificates and get the hosts communicating with the ovirt-engine
>>> >>> >> >> again so that I can bring the data center back online.
>>> >>> >> >>
>>> >>> >> >> I found this page which deals with changing the engine hostname,
>>> >>> >> >> and
>>> >>> >> >> thus re-creating the certificates and keystore on the ovirt-engine
>>> >>> >> >> node, and was wondering if this could help. Could I follow this
>>> >>> >> >> process but keep the same hostname for the ovirt-engine node?
>>> >>> >> >>
>>> >>> >> >> http://wiki.ovirt.org/How_to_change_engine_host_name
>>> >>> >> >>
>>> >>> >> >> Currently I have 3 VM's running on two hosts. The VM's are up, but
>>> >>> >> >> I
>>> >>> >> >> can't do anything with them in ovirt-engine.
>>> >>> >> >>
>>> >>> >> >>
>>> >>> >> >> Here's the latest activity from engine.log from the ovirt-engine
>>> >>> >> >> node:
>>> >>> >> >>
>>> >>> >> >> 2013-04-06 21:58:47,472 ERROR
>>> >>> >> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>> >>> >> >> (QuartzScheduler_Worker-61) Failed to
>>> >>> >> >> decryptjava.io.FileNotFoundException:
>>> >>> >> >> /etc/pki/ovirt-engine/.keystore
>>> >>> >> >> (Permission denied)
>>> >>> >> >> 2013-04-06 21:58:47,478 ERROR
>>> >>> >> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>> >>> >> >> (QuartzScheduler_Worker-62) Can't load keystore from file
>>> >>> >> >> "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException:
>>> >>> >> >> /etc/pki/ovirt-engine/.keystore (Permission denied)
>>> >>> >> >> at java.io.FileInputStream.open(Native Method)
>>> >>> >> >> [rt.jar:1.7.0_09-icedtea]
>>> >>> >> >> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>> >>> >> >> [rt.jar:1.7.0_09-icedtea]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214)
>>> >>> >> >> [engine-encryptutils.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139)
>>> >>> >> >> [engine-encryptutils.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164)
>>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31)
>>> >>> >> >> [engine-dal.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219)
>>> >>> >> >> [engine-vdsbroker.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168)
>>> >>> >> >> [engine-utils.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107)
>>> >>> >> >> [engine-utils.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215)
>>> >>> >> >> [engine-vdsbroker.jar:]
>>> >>> >> >> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown
>>> >>> >> >> Source) [:1.7.0_09-icedtea]
>>> >>> >> >> at
>>> >>> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >>> >> >> [rt.jar:1.7.0_09-icedtea]
>>> >>> >> >> at java.lang.reflect.Method.invoke(Method.java:601)
>>> >>> >> >> [rt.jar:1.7.0_09-icedtea]
>>> >>> >> >> at
>>> >>> >> >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64)
>>> >>> >> >> [engine-scheduler.jar:]
>>> >>> >> >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>> >>> >> >> [quartz.jar:]
>>> >>> >> >> at
>>> >>> >> >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>>> >>> >> >> [quartz.jar:]
>>> >>> >> >>
>>> >>> >> >> 2013-04-06 21:58:47,576 ERROR
>>> >>> >> >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>> >>> >> >> (QuartzScheduler_Worker-61) XML RPC error in command
>>> >>> >> >> GetCapabilitiesVDS ( Vds: defiant ), the error was:
>>> >>> >> >> java.util.concurrent.ExecutionException:
>>> >>> >> >> java.lang.reflect.InvocationTargetException,
>>> >>> >> >> SSLPeerUnverifiedException: peer not authenticated
>>> >>> >> >> 2013-04-06 21:58:47,606 ERROR
>>> >>> >> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>> >>> >> >> (QuartzScheduler_Worker-62) Failed to
>>> >>> >> >> decryptjava.io.FileNotFoundException:
>>> >>> >> >> /etc/pki/ovirt-engine/.keystore
>>> >>> >> >> (Permission denied)
>>> >>> >> >> 2013-04-06 21:58:47,671 ERROR
>>> >>> >> >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>> >>> >> >> (QuartzScheduler_Worker-62) XML RPC error in command
>>> >>> >> >> GetCapabilitiesVDS ( Vds: transporter ), the error was:
>>> >>> >> >> java.util.concurrent.ExecutionException:
>>> >>> >> >> java.lang.reflect.InvocationTargetException,
>>> >>> >> >> SSLPeerUnverifiedException: peer not authenticated
>>> >>> >> >>
>>> >>> >> >>
>>> >>> >> >> Here's the message I seem to get over and over on the fedora 17
>>> >>> >> >> host in
>>> >>> >> >> vdsm.log
>>> >>> >> >>
>>> >>> >> >> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>>> >>> >> >> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>> >>> >> >> Thread-562520::ERROR::2013-04-06
>>> >>> >> >> 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client
>>> >>> >> >> ('172.16.23.8', 36127)
>>> >>> >> >> Traceback (most recent call last):
>>> >>> >> >> File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>>> >>> >> >> process_request_thread
>>> >>> >> >> self.finish_request(request, client_address)
>>> >>> >> >> File
>>> >>> >> >> "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>>> >>> >> >> line 66, in finish_request
>>> >>> >> >> request.do_handshake()
>>> >>> >> >> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>>> >>> >> >> self._sslobj.do_handshake()
>>> >>> >> >>
>>> >>> >> >> I'm also wondering about the permission denied on the .keystore
>>> >>> >> >> directory. What should the permissions be? Here's what they are
>>> >>> >> >> currently.
>>> >>> >> >>
>>> >>> >> >> [root at reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore
>>> >>> >> >> -rwxr-x---. root root unconfined_u:object_r:cert_t:s0
>>> >>> >> >> /etc/pki/ovirt-engine/.keystore
>>> >>> >> >>
>>> >>> >> >> I also seem to have a backup of the ovirt-engine directory at the
>>> >>> >> >> time
>>> >>> >> >> the update was performed, but replacing ovirt-engine with the
>>> >>> >> >> backup
>>> >>> >> >> does no good.
>>> >>> >> >>
>>> >>> >> >> I appreciate any assistance, and please let me know what other
>>> >>> >> >> information I can post to help with this.
>>> >>> >> >>
>>> >>> >> >> Thanks
>>> >>> >> >> _______________________________________________
>>> >>> >> >> Users mailing list
>>> >>> >> >> Users at ovirt.org
>>> >>> >> >> http://lists.ovirt.org/mailman/listinfo/users
>>> >>> >> >>
>>> >>> >>
>>> >>>
>>>
More information about the Users
mailing list