Re: [Users] Certificates and PKI seem to be broken after yum update

engine.log attached On Thu, Apr 18, 2013 at 7:11 PM, Alon Bar-Lev <alonbl(a)redhat.com&gt; wrote: > Need to know precise error, please attach engine.log. > > > ----- Original Message ----- >> From: "Chris Smith" <whitehat237(a)gmail.com&gt; >> To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt; >> Cc: Users(a)ovirt.org >> Sent: Friday, April 19, 2013 2:03:59 AM >> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update >> >> So as of now, I can put the host into maintenance mode using the >> ovirt-engine web interface. I can also try and activate it. It >> states that the host was activated. The host never actually comes up >> or contends for SPM status, and the data center never actually comes >> online. >> >> If I put the host into maintenance mode and try to reinstall it, it >> throws an error and size must be between 0 and 50. >> >> On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev <alonbl(a)redhat.com&gt; wrote: >> > I am not sure I understand the status. >> > >> > Everything is working or not. >> > If not, what exactly fails? >> > Why do you run it 'again'? >> > >> > What happens if you reinstall host? Go to maintenance and select reinstall? >> > >> > I cannot understand how all this results from upgrade, something had >> > changed, the CA certificate installed on the host is probably not the CA >> > certificate of the engine. >> > >> > ----- Original Message ----- >> >> From: "Chris Smith" <whitehat237(a)gmail.com&gt; >> >> To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt;, Users(a)ovirt.org >> >> Sent: Friday, April 19, 2013 1:45:23 AM >> >> Subject: Re: [Users] Certificates and PKI seem to be broken after yum >> >> update >> >> >> >> On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith <whitehat237(a)gmail.com&gt; >> >> wrote: >> >> > I made a backup of the .truststore, and then followed the steps and >> >> > then rebooted both the ovirt-engine and one of the hosts, and >> >> > everything worked properly. >> >> > >> >> > If I run it again, or enter the wrong password it throws an error >> >> > about the key store already existing, or that the password was wrong >> >> > so I'm pretty sure it's good. >> >> > >> >> > vdsm.log on the host still shows: >> >> > >> >> > Traceback (most recent call last): >> >> > File "/usr/lib64/python2.7/SocketServer.py", line 582, in >> >> > process_request_thread >> >> > self.finish_request(request, client_address) >> >> > File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", >> >> > line 66, in finish_request >> >> > request.do_handshake() >> >> > File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake >> >> > self._sslobj.do_handshake() >> >> > SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL >> >> > routines:SSL3_READ_BYTES:sslv3 alert certificate unknown >> >> > >> >> > engine.log on the host shows: >> >> > >> >> > 2013-04-18 18:42:43,632 ERROR >> >> > [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> >> > (QuartzScheduler_Worker-68) Failed to decryptData must start with zero >> >> > 2013-04-18 18:42:43,642 ERROR >> >> > [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] >> >> > (QuartzScheduler_Worker-68) XML RPC error in command >> >> > GetCapabilitiesVDS ( Vds: transporter ), the error was: >> >> > java.util.concurrent.ExecutionException: >> >> > java.lang.reflect.InvocationTargetException, >> >> > SunCertPathBuilderException: unable to find valid certification path >> >> > to requested target >> >> > >> >> > >> >> > On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev <alonbl(a)redhat.com&gt; wrote: >> >> >> >> >> >> You should ask these question in separate thread so people may pick >> >> >> them >> >> >> up. >> >> >> >> >> >> For the .truststore, try to remove it and then execute: >> >> >> >> >> >> # rm -f /etc/pki/ovirt-engine/.truststore >> >> >> # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass >> >> >> -file /etc/pki/ovirt-engine/certs/ca.der -keystore >> >> >> /etc/pki/ovirt-engine/.truststore -storepass mypass >> >> >> # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore >> >> >> >> >> >> It should recreate the truststore with the ca certificate you have. >> >> >> >> >> >> ----- Original Message ----- >> >> >>> From: "Chris Smith" <whitehat237(a)gmail.com&gt; >> >> >>> To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt; >> >> >>> Cc: Users(a)ovirt.org >> >> >>> Sent: Thursday, April 18, 2013 7:18:27 AM >> >> >>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum >> >> >>> update >> >> >>> >> >> >>> If it would be easier than re-setting up the certificates, I'm also >> >> >>> willing to just start over and rebuild, but I would like to export the >> >> >>> VM's I have first. >> >> >>> One of them is a spacewalk server, another runs DNS, and DHCP for my >> >> >>> test network, and I have an asterisk server. I would like to avoid >> >> >>> having to re-create all of them. >> >> >>> >> >> >>> The VM's are up and running now, so I could export all of the >> >> >>> configurations / backup the file systems, etc. >> >> >>> >> >> >>> Preferably I could export the VM's to an NFS export domain, or a >> >> >>> mounted NFS share so that I can import them to the new storage domain, >> >> >>> after I run engine-cleanup and get everything set back up. Is there >> >> >>> an easy way to do this? Is it possible to create and attach an NFS >> >> >>> export domain directly from the CLI without access to the ovirt >> >> >>> manager without communication between the manager and hosts due to the >> >> >>> pki issue? Can I export the VM's directly from the hosts to a >> >> >>> standard NFS share? >> >> >>> >> >> >>> Is there an equivalent xml and image file for the VM? >> >> >>> >> >> >>> My storage domain is iscsi and is served out from another server over >> >> >>> 4 bonded 1 Gbps copper links. >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith <whitehat237(a)gmail.com&gt; >> >> >>> wrote: >> >> >>> > I checked the .truststore on the ovirt engine, and it seems fine. >> >> >>> > >> >> >>> > [root@reliant ovirt-engine]# ls -l .truststore >> >> >>> > -rwxr-x---. 1 ovirt ovirt 918 Apr 6 21:56 .truststore >> >> >>> > >> >> >>> > It's not zero bytes anyway. >> >> >>> > >> >> >>> > It's also the same size as the .truststore in the ovirt engine >> >> >>> > backups. >> >> >>> > >> >> >>> > [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec >> >> >>> > ls >> >> >>> > -l >> >> >>> > {} \; >> >> >>> > -rwxr-x---. 1 ovirt ovirt 918 Aug 26 2012 >> >> >>> > ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore >> >> >>> > -rwxr-x---. 1 root root 918 Mar 24 12:42 >> >> >>> > ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore >> >> >>> > >> >> >>> > I haven't looked at the installCA.sh script yet. >> >> >>> > >> >> >>> > On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev <alonbl(a)redhat.com&gt; >> >> >>> > wrote: >> >> >>> >> This error means that the /etc/pki/ovirt-engine/.truststore is >> >> >>> >> unreadable >> >> >>> >> or does not contain the /etc/pki/ovirt-engine/ca.pem certificate. >> >> >>> >> >> >> >>> >> Unfortunately, the pki administration is weak in current >> >> >>> >> implementation, >> >> >>> >> you can trace the installation script and checkout the calls to >> >> >>> >> installCA.sh to how to reproduce, please note that password are >> >> >>> >> encrypted >> >> >>> >> in database using the private key locate in .keystore so if you are >> >> >>> >> to >> >> >>> >> re-generate anything remember to keep the engine private key. >> >> >>> >> >> >> >>> >> However, if you succeed in login, the remaining problem you have is >> >> >>> >> the >> >> >>> >> .truststore permissions and/or content. >> >> >>> >> >> >> >>> >> Regards, >> >> >>> >> Alon Bar-Lev. >> >> >>> >> >> >> >>> >> ----- Original Message ----- >> >> >>> >>> From: "Chris Smith" <whitehat237(a)gmail.com&gt; >> >> >>> >>> To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt; >> >> >>> >>> Cc: Users(a)ovirt.org >> >> >>> >>> Sent: Monday, April 8, 2013 9:46:46 AM >> >> >>> >>> Subject: Re: [Users] Certificates and PKI seem to be broken after >> >> >>> >>> yum >> >> >>> >>> update >> >> >>> >>> >> >> >>> >>> After setting the .keystore owner and group owner to ovirt, and >> >> >>> >>> rebooting, I now have a new error in engine.log >> >> >>> >>> >> >> >>> >>> 2013-04-08 02:39:16,787 ERROR >> >> >>> >>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> >> >>> >>> (QuartzScheduler_Worker-95) Failed to decryptData must start with >> >> >>> >>> zero >> >> >>> >>> 2013-04-08 02:39:16,845 ERROR >> >> >>> >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] >> >> >>> >>> (QuartzScheduler_Worker-95) XML RPC error in command >> >> >>> >>> GetCapabilitiesVDS ( Vds: transporter ), the error was: >> >> >>> >>> java.util.concurrent.ExecutionException: >> >> >>> >>> java.lang.reflect.InvocationTargetException, >> >> >>> >>> SunCertPathBuilderException: unable to find valid certification >> >> >>> >>> path >> >> >>> >>> to requested target >> >> >>> >>> >> >> >>> >>> Are there other files that may have been affected that I can also >> >> >>> >>> correct ownership or permissions on? >> >> >>> >>> >> >> >>> >>> On the host side, I get certificate unknown in vdsm.log >> >> >>> >>> >> >> >>> >>> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake >> >> >>> >>> self._sslobj.do_handshake() >> >> >>> >>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL >> >> >>> >>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown >> >> >>> >>> Thread-757809::ERROR::2013-04-08 >> >> >>> >>> 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client >> >> >>> >>> ('172.16.23.8', 54489) >> >> >>> >>> Traceback (most recent call last): >> >> >>> >>> File "/usr/lib64/python2.7/SocketServer.py", line 582, in >> >> >>> >>> process_request_thread >> >> >>> >>> self.finish_request(request, client_address) >> >> >>> >>> File >> >> >>> >>> "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", >> >> >>> >>> line 66, in finish_request >> >> >>> >>> request.do_handshake() >> >> >>> >>> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake >> >> >>> >>> self._sslobj.do_handshake() >> >> >>> >>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL >> >> >>> >>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown >> >> >>> >>> >> >> >>> >>> Is there a procedure for just re-establishing PKI and certs for >> >> >>> >>> the >> >> >>> >>> engine and hosts? >> >> >>> >>> >> >> >>> >>> On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev <alonbl(a)redhat.com&gt; >> >> >>> >>> wrote: >> >> >>> >>> > >> >> >>> >>> > OK... you are running a very old version of engine (3.1). >> >> >>> >>> > >> >> >>> >>> > The upgrade did not upgraded into 3.2, so nothing as far as I >> >> >>> >>> > know >> >> >>> >>> > should >> >> >>> >>> > have been changed. >> >> >>> >>> > >> >> >>> >>> > But the .keystore permissions is owned by root now, so some >> >> >>> >>> > other >> >> >>> >>> > package >> >> >>> >>> > (maybe selinux-policy) changed permissions... >> >> >>> >>> > >> >> >>> >>> > The simplest way to test is to: >> >> >>> >>> > # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 >> >> >>> >>> > # chown -R ovirt:ovirt /etc/pki/ovirt-engine >> >> >>> >>> > >> >> >>> >>> > But if that file permissions was changed, I can only assume >> >> >>> >>> > other >> >> >>> >>> > files >> >> >>> >>> > were also changes... >> >> >>> >>> > >> >> >>> >>> > Regards, >> >> >>> >>> > Alon >> >> >>> >>> > >> >> >>> >>> > ----- Original Message ----- >> >> >>> >>> >> From: "Chris Smith" <whitehat237(a)gmail.com&gt; >> >> >>> >>> >> To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt; >> >> >>> >>> >> Cc: Users(a)ovirt.org >> >> >>> >>> >> Sent: Sunday, April 7, 2013 11:51:17 AM >> >> >>> >>> >> Subject: Re: [Users] Certificates and PKI seem to be broken >> >> >>> >>> >> after >> >> >>> >>> >> yum >> >> >>> >>> >> update >> >> >>> >>> >> >> >> >>> >>> >> I did a yum update and rebooted. >> >> >>> >>> >> >> >> >>> >>> >> engine-upgrade was run on 24-March >> >> >>> >>> >> >> >> >>> >>> >> When run now, it states that there are no updates available. >> >> >>> >>> >> >> >> >>> >>> >> [root@reliant ~]# engine-upgrade >> >> >>> >>> >> Loaded plugins: versionlock >> >> >>> >>> >> Checking for updates... (This may take several minutes) >> >> >>> >>> >> No updates available >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> [root@reliant ovirt-engine]# cat >> >> >>> >>> >> ovirt-engine-upgrade_2013_03_24_12_04_06.log >> >> >>> >>> >> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found >> >> >>> >>> >> existing >> >> >>> >>> >> pgpass file, fetching DB host value >> >> >>> >>> >> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found >> >> >>> >>> >> existing >> >> >>> >>> >> pgpass file, fetching DB port value >> >> >>> >>> >> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found >> >> >>> >>> >> existing >> >> >>> >>> >> pgpass file, fetching DB admin value >> >> >>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum >> >> >>> >>> >> list >> >> >>> >>> >> updates >> >> >>> >>> >> started >> >> >>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum >> >> >>> >>> >> unlock >> >> >>> >>> >> started >> >> >>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum >> >> >>> >>> >> unlock >> >> >>> >>> >> completed successfully >> >> >>> >>> >> 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting >> >> >>> >>> >> list >> >> >>> >>> >> of packages to upgrade >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum >> >> >>> >>> >> lock >> >> >>> >>> >> started >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-backend' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-backend-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-config' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-config-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-genericapi' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-genericapi-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-notification-service' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-notification-service-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-restapi' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-restapi-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-tools-common' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-tools-common-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-userportal' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-userportal-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing >> >> >>> >>> >> command --> '/bin/rpm -q ovirt-engine-webadmin-portal' >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = >> >> >>> >>> >> ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch >> >> >>> >>> >> >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = >> >> >>> >>> >> /bin/rpm >> >> >>> >>> >> -q ovirt-engine ovirt-engine-backend ovirt-engine-config >> >> >>> >>> >> ovirt-engine-genericapi ovirt-engine-notification-service >> >> >>> >>> >> ovirt-engine-restapi ovirt-engine-tools-common >> >> >>> >>> >> ovirt-engine-userportal >> >> >>> >>> >> ovirt-engine-webadmin-portal >> >> >> >>> >>> >> /etc/yum/pluginconf.d/versionlock.list >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = >> >> >>> >>> >> 0 >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum >> >> >>> >>> >> lock >> >> >>> >>> >> completed successfully >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No >> >> >>> >>> >> packages >> >> >>> >>> >> marked for update >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: >> >> >>> >>> >> Installed >> >> >>> >>> >> packages: >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: >> >> >>> >>> >> ['ovirt-engine-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-backend-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-config-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-setup-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', >> >> >>> >>> >> 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', >> >> >>> >>> >> 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', >> >> >>> >>> >> 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', >> >> >>> >>> >> 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum >> >> >>> >>> >> list >> >> >>> >>> >> updated completed successfully >> >> >>> >>> >> 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No >> >> >>> >>> >> updates >> >> >>> >>> >> available >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> Here's what's installed. >> >> >>> >>> >> >> >> >>> >>> >> [root@reliant yum.repos.d]# yum list installed | grep ovirt >> >> >>> >>> >> ovirt-engine.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-backend.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-cli.noarch 3.2.0.5-1.fc17 >> >> >>> >>> >> @updates >> >> >>> >>> >> ovirt-engine-config.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-genericapi.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-notification-service.noarch >> >> >>> >>> >> 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-restapi.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 >> >> >>> >>> >> @updates >> >> >>> >>> >> ovirt-engine-setup.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-tools-common.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-userportal.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 >> >> >>> >>> >> @ovirt-stable >> >> >>> >>> >> ovirt-release-fedora.noarch 4-2 >> >> >>> >>> >> @/ovirt-release-fedora.noarch >> >> >>> >>> >> >> >> >>> >>> >> On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev >> >> >>> >>> >> <alonbl(a)redhat.com&gt; >> >> >>> >>> >> wrote: >> >> >>> >>> >> > How exactly did you upgrade? >> >> >>> >>> >> > >> >> >>> >>> >> > Usually yum upgrade will not touch ovirt-engine packages as >> >> >>> >>> >> > it >> >> >>> >>> >> > is in >> >> >>> >>> >> > yum >> >> >>> >>> >> > version lock. >> >> >>> >>> >> > From which version to which version have you upgraded? >> >> >>> >>> >> > Have you run engine-upgrade utility? >> >> >>> >>> >> > If you did not, please run it. >> >> >>> >>> >> > If you did, please attach logs from >> >> >>> >>> >> > /var/log/ovirt-engine/ovirt-engine-upgrade* >> >> >>> >>> >> > >> >> >>> >>> >> > Thanks! >> >> >>> >>> >> > >> >> >>> >>> >> > ----- Original Message ----- >> >> >>> >>> >> >> From: "Chris Smith" <whitehat237(a)gmail.com&gt; >> >> >>> >>> >> >> To: Users(a)ovirt.org >> >> >>> >>> >> >> Sent: Sunday, April 7, 2013 5:09:46 AM >> >> >>> >>> >> >> Subject: [Users] Certificates and PKI seem to be broken >> >> >>> >>> >> >> after >> >> >>> >>> >> >> yum >> >> >>> >>> >> >> update >> >> >>> >>> >> >> >> >> >>> >>> >> >> I have lost the ability to manage the hosts or VM's using >> >> >>> >>> >> >> ovirt >> >> >>> >>> >> >> engine web interface after performing yum update on the >> >> >>> >>> >> >> ovirt-engine >> >> >>> >>> >> >> host, and on one Fedora 17 host. The data center is >> >> >>> >>> >> >> offline, >> >> >>> >>> >> >> and I >> >> >>> >>> >> >> can't place the hosts into maintenance mode. I don't think >> >> >>> >>> >> >> that >> >> >>> >>> >> >> there >> >> >>> >>> >> >> are any actions I can perform in the web interface at all. >> >> >>> >>> >> >> >> >> >>> >>> >> >> From the logs it seems that PKI is broken between the engine >> >> >>> >>> >> >> and >> >> >>> >>> >> >> the >> >> >>> >>> >> >> hosts. >> >> >>> >>> >> >> >> >> >>> >>> >> >> I am wondering how I can restore or re-generate all of the >> >> >>> >>> >> >> certificates and get the hosts communicating with the >> >> >>> >>> >> >> ovirt-engine >> >> >>> >>> >> >> again so that I can bring the data center back online. >> >> >>> >>> >> >> >> >> >>> >>> >> >> I found this page which deals with changing the engine >> >> >>> >>> >> >> hostname, >> >> >>> >>> >> >> and >> >> >>> >>> >> >> thus re-creating the certificates and keystore on the >> >> >>> >>> >> >> ovirt-engine >> >> >>> >>> >> >> node, and was wondering if this could help. Could I follow >> >> >>> >>> >> >> this >> >> >>> >>> >> >> process but keep the same hostname for the ovirt-engine >> >> >>> >>> >> >> node? >> >> >>> >>> >> >> >> >> >>> >>> >> >> http://wiki.ovirt.org/How_to_change_engine_host_name >> >> >>> >>> >> >> >> >> >>> >>> >> >> Currently I have 3 VM's running on two hosts. The VM's are >> >> >>> >>> >> >> up, >> >> >>> >>> >> >> but >> >> >>> >>> >> >> I >> >> >>> >>> >> >> can't do anything with them in ovirt-engine. >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> Here's the latest activity from engine.log from the >> >> >>> >>> >> >> ovirt-engine >> >> >>> >>> >> >> node: >> >> >>> >>> >> >> >> >> >>> >>> >> >> 2013-04-06 21:58:47,472 ERROR >> >> >>> >>> >> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> >> >>> >>> >> >> (QuartzScheduler_Worker-61) Failed to >> >> >>> >>> >> >> decryptjava.io.FileNotFoundException: >> >> >>> >>> >> >> /etc/pki/ovirt-engine/.keystore >> >> >>> >>> >> >> (Permission denied) >> >> >>> >>> >> >> 2013-04-06 21:58:47,478 ERROR >> >> >>> >>> >> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> >> >>> >>> >> >> (QuartzScheduler_Worker-62) Can't load keystore from file >> >> >>> >>> >> >> "/etc/pki/ovirt-engine/.keystore".: >> >> >>> >>> >> >> java.io.FileNotFoundException: >> >> >>> >>> >> >> /etc/pki/ovirt-engine/.keystore (Permission denied) >> >> >>> >>> >> >> at java.io.FileInputStream.open(Native Method) >> >> >>> >>> >> >> [rt.jar:1.7.0_09-icedtea] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> java.io.FileInputStream.<init>(FileInputStream.java:138) >> >> >>> >>> >> >> [rt.jar:1.7.0_09-icedtea] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) >> >> >>> >>> >> >> [engine-encryptutils.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) >> >> >>> >>> >> >> [engine-encryptutils.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) >> >> >>> >>> >> >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) >> >> >>> >>> >> >> [engine-dal.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) >> >> >>> >>> >> >> [engine-vdsbroker.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) >> >> >>> >>> >> >> [engine-utils.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) >> >> >>> >>> >> >> [engine-utils.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) >> >> >>> >>> >> >> [engine-vdsbroker.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> sun.reflect.GeneratedMethodAccessor13.invoke(Unknown >> >> >>> >>> >> >> Source) [:1.7.0_09-icedtea] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> >>> >>> >> >> [rt.jar:1.7.0_09-icedtea] >> >> >>> >>> >> >> at java.lang.reflect.Method.invoke(Method.java:601) >> >> >>> >>> >> >> [rt.jar:1.7.0_09-icedtea] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) >> >> >>> >>> >> >> [engine-scheduler.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> >> >>> >>> >> >> [quartz.jar:] >> >> >>> >>> >> >> at >> >> >>> >>> >> >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> >> >>> >>> >> >> [quartz.jar:] >> >> >>> >>> >> >> >> >> >>> >>> >> >> 2013-04-06 21:58:47,576 ERROR >> >> >>> >>> >> >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] >> >> >>> >>> >> >> (QuartzScheduler_Worker-61) XML RPC error in command >> >> >>> >>> >> >> GetCapabilitiesVDS ( Vds: defiant ), the error was: >> >> >>> >>> >> >> java.util.concurrent.ExecutionException: >> >> >>> >>> >> >> java.lang.reflect.InvocationTargetException, >> >> >>> >>> >> >> SSLPeerUnverifiedException: peer not authenticated >> >> >>> >>> >> >> 2013-04-06 21:58:47,606 ERROR >> >> >>> >>> >> >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] >> >> >>> >>> >> >> (QuartzScheduler_Worker-62) Failed to >> >> >>> >>> >> >> decryptjava.io.FileNotFoundException: >> >> >>> >>> >> >> /etc/pki/ovirt-engine/.keystore >> >> >>> >>> >> >> (Permission denied) >> >> >>> >>> >> >> 2013-04-06 21:58:47,671 ERROR >> >> >>> >>> >> >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] >> >> >>> >>> >> >> (QuartzScheduler_Worker-62) XML RPC error in command >> >> >>> >>> >> >> GetCapabilitiesVDS ( Vds: transporter ), the error was: >> >> >>> >>> >> >> java.util.concurrent.ExecutionException: >> >> >>> >>> >> >> java.lang.reflect.InvocationTargetException, >> >> >>> >>> >> >> SSLPeerUnverifiedException: peer not authenticated >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> Here's the message I seem to get over and over on the fedora >> >> >>> >>> >> >> 17 >> >> >>> >>> >> >> host in >> >> >>> >>> >> >> vdsm.log >> >> >>> >>> >> >> >> >> >>> >>> >> >> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL >> >> >>> >>> >> >> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown >> >> >>> >>> >> >> Thread-562520::ERROR::2013-04-06 >> >> >>> >>> >> >> 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) >> >> >>> >>> >> >> client >> >> >>> >>> >> >> ('172.16.23.8', 36127) >> >> >>> >>> >> >> Traceback (most recent call last): >> >> >>> >>> >> >> File "/usr/lib64/python2.7/SocketServer.py", line 582, in >> >> >>> >>> >> >> process_request_thread >> >> >>> >>> >> >> self.finish_request(request, client_address) >> >> >>> >>> >> >> File >> >> >>> >>> >> >> "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", >> >> >>> >>> >> >> line 66, in finish_request >> >> >>> >>> >> >> request.do_handshake() >> >> >>> >>> >> >> File "/usr/lib64/python2.7/ssl.py", line 305, in >> >> >>> >>> >> >> do_handshake >> >> >>> >>> >> >> self._sslobj.do_handshake() >> >> >>> >>> >> >> >> >> >>> >>> >> >> I'm also wondering about the permission denied on the >> >> >>> >>> >> >> .keystore >> >> >>> >>> >> >> directory. What should the permissions be? Here's what >> >> >>> >>> >> >> they >> >> >>> >>> >> >> are >> >> >>> >>> >> >> currently. >> >> >>> >>> >> >> >> >> >>> >>> >> >> [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore >> >> >>> >>> >> >> -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 >> >> >>> >>> >> >> /etc/pki/ovirt-engine/.keystore >> >> >>> >>> >> >> >> >> >>> >>> >> >> I also seem to have a backup of the ovirt-engine directory >> >> >>> >>> >> >> at >> >> >>> >>> >> >> the >> >> >>> >>> >> >> time >> >> >>> >>> >> >> the update was performed, but replacing ovirt-engine with >> >> >>> >>> >> >> the >> >> >>> >>> >> >> backup >> >> >>> >>> >> >> does no good. >> >> >>> >>> >> >> >> >> >>> >>> >> >> I appreciate any assistance, and please let me know what >> >> >>> >>> >> >> other >> >> >>> >>> >> >> information I can post to help with this. >> >> >>> >>> >> >> >> >> >>> >>> >> >> Thanks >> >> >>> >>> >> >> _______________________________________________ >> >> >>> >>> >> >> Users mailing list >> >> >>> >>> >> >> Users(a)ovirt.org >> >> >>> >>> >> >> http://lists.ovirt.org/mailman/listinfo/users >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >>> >>> >> >> >>> >> >> >>