Hi Alon,
That has sorted it out. The permissions got messed up in between
restoring from previous backups etc.
Thank you very much, greatly appreciated.
Regards.
Neil Wilson
On Wed, May 28, 2014 at 9:11 AM, Alon Bar-Lev <alonbl(a)redhat.com> wrote:
----- Original Message -----
> From: "Neil" <nwilson123(a)gmail.com>
> To: "Alon Bar-Lev" <alonbl(a)redhat.com>
> Cc: users(a)ovirt.org
> Sent: Wednesday, May 28, 2014 10:04:00 AM
> Subject: Re: [ovirt-users] Can't Install/Upgrade host
>
> Hi Alon,
>
> Thanks for the reply, below is the output.
Something changed the file attributes of ca.pem (two places) to be incorrect.
> [root@engine01 ovirt-engine]# ls -lR /etc/pki/ovirt-engine/
> /etc/pki/ovirt-engine/:
> total 80
> lrwxrwxrwx. 1 root root 6 May 16 13:56 apache-ca.pem -> ca.pem
> -rw-r--r--. 1 root root 570 May 16 13:56 cacert.conf
> -rw-r--r--. 1 root root 519 May 16 13:56 cacert.template
> -rw-r--r--. 1 root root 384 Mar 24 12:47 cacert.template.in
> -rw-r--r--. 1 root root 482 May 16 13:56 cacert.template.rpmnew
> -rwxr-x---. 1 root root 3362 May 16 13:56 ca.pem
this ^ should be world readable, not executable.
> -rw-r--r--. 1 root root 585 May 16 13:56 cert.conf
> drwxr-xr-x. 2 ovirt ovirt 4096 Mar 24 12:47 certs
> -rw-r--r--. 1 root root 572 May 16 13:56 cert.template
> -rw-r--r--. 1 root root 483 Mar 24 12:47 cert.template.in
> -rw-r--r--. 1 root root 534 May 16 13:56 cert.template.rpmnew
> -rw-r--r--. 1 ovirt ovirt 950 May 22 20:07 database.txt
> -rw-r--r--. 1 ovirt ovirt 20 May 22 20:07 database.txt.attr
> -rw-r--r--. 1 ovirt ovirt 20 May 16 13:56 database.txt.attr.old
> -rw-r--r--. 1 ovirt ovirt 885 May 16 13:56 database.txt.old
> drwxr-xr-x. 2 root root 4096 Mar 24 12:47 keys
> -rw-r--r--. 1 root root 548 Mar 24 12:47 openssl.conf
> drwxr-x---. 2 ovirt ovirt 4096 Mar 24 12:47 private
> drwxr-xr-x. 2 ovirt ovirt 4096 May 27 13:16 requests
> -rw-r--r--. 1 ovirt ovirt 3 May 22 20:07 serial.txt
> -rw-r--r--. 1 ovirt ovirt 3 May 16 13:56 serial.txt.old
>
> /etc/pki/ovirt-engine/certs:
> total 100
> -rw-r--r--. 1 root root 3362 May 16 13:56 01.pem
> -rw-r--r--. 1 root root 3509 May 16 13:56 02.pem
> -rw-r--r--. 1 root root 3466 May 16 13:56 03.pem
> -rw-r--r--. 1 root root 3466 May 16 13:56 04.pem
> -rw-r--r--. 1 root root 3362 May 16 13:56 05.pem
> -rw-r--r--. 1 root root 3509 May 16 13:56 06.pem
> -rw-r--r--. 1 root root 3362 May 16 13:56 07.pem
> -rw-r--r--. 1 root root 3509 May 16 13:56 08.pem
> -rw-r--r--. 1 root root 3466 May 16 13:56 09.pem
> -rw-r--r--. 1 root root 3467 May 16 13:56 0A.pem
> -rw-r--r--. 1 root root 3467 May 16 13:56 0B.pem
> -rw-r--r--. 1 root root 3467 May 16 13:56 0C.pem
> -rw-r--r--. 1 root root 3467 May 16 13:56 0D.pem
> -rw-r--r--. 1 root root 3070 May 16 13:56 0E.pem
> -rw-r--r--. 1 root root 3070 May 16 13:56 0F.pem
> -rw-r--r--. 1 root root 3070 May 16 13:56 10.251.193.8cert.pem
> -rw-r--r--. 1 root root 3070 May 16 13:56 10.251.193.9cert.pem
these two are strange as I expect to be owned by ovirt user as engine created.
> -rw-r--r--. 1 root root 4267 May 22 20:07 10.pem
> -rw-r-----. 1 root root 3509 May 16 13:56 apache.cer
> -rw-r--r--. 1 root root 763 May 16 13:56 ca.der
> -rw-r--r--. 1 root root 3509 May 16 13:56 engine.cer
> -rw-r--r--. 1 root root 784 May 16 13:56 engine.der
> -rw-r--r--. 1 root root 4267 May 22 20:07 websocket-proxy.cer
>
> /etc/pki/ovirt-engine/keys:
> total 36
> -rw-r-----. 1 root root 916 May 16 13:56 apache.key.nopass
> -rw-r-----. 1 root root 2786 May 16 13:56 apache.p12
> -rw-------. 1 root root 1054 May 22 20:07 engine_id_rsa
> -rw-------. 1 root root 916 May 16 13:56 engine_id_rsa.20140522200739
> -rw-------. 1 root root 912 May 16 13:56 engine_id_rsa.old
> -rw-r-----. 1 ovirt ovirt 2786 May 16 13:56 engine.p12
> -rw-r--r--. 1 root root 220 May 16 13:56 engine.ssh.key.txt
> -rw-------. 1 ovirt ovirt 1832 May 22 20:07 websocket-proxy.key.nopass
> -rw-------. 1 root root 2517 May 22 20:07 websocket-proxy.p12
>
> /etc/pki/ovirt-engine/private:
> total 4
> -rwxr-x---. 1 root root 887 May 16 13:56 ca.pem
this should be owned by ovirt user and not be executable.
>
> /etc/pki/ovirt-engine/requests:
> total 24
> -rw-r--r--. 1 root root 862 May 16 13:56 10.251.193.8req.pem
> -rw-r--r--. 1 ovirt ovirt 862 May 27 17:35 10.251.193.9.req
> -rw-r--r--. 1 root root 862 May 16 13:56 10.251.193.9req.pem
> -rw-r--r--. 1 root root 603 May 16 13:56 ca.csr
> -rw-r--r--. 1 root root 597 May 16 13:56 engine.req
> -rw-r--r--. 1 root root 863 May 22 20:07 websocket-proxy.req
>
>
>
> On Wed, May 28, 2014 at 8:19 AM, Alon Bar-Lev <alonbl(a)redhat.com> wrote:
> > Please send the output of:
> >
> > # ls -lR /etc/pki/ovirt-engine/
> >
> > ----- Original Message -----
> >> From: "Neil" <nwilson123(a)gmail.com>
> >> To: users(a)ovirt.org
> >> Sent: Wednesday, May 28, 2014 9:04:57 AM
> >> Subject: [ovirt-users] Can't Install/Upgrade host
> >>
> >> Hi guys,
> >>
> >> I'm trying to upgrade/re-install a host running Centos 6.5, but even
> >> after removing the host completely and trying to re-add it, I keep
> >> getting a "Certificate enrollment failed" error. The full error
below
> >> is taken from my engine.log...
> >>
> >> 2014-05-27 10:38:33,729 ERROR
> >> [org.ovirt.engine.core.utils.servlet.ServletUtils]
> >> (ajp--127.0.0.1-8702-4) Can't read file
> >> "/var/lib/ovirt-engine/reports.xml" for request
> >> "/ovirt-engine/services/reports-ui", will send a 404 error
response.
> >> 2014-05-27 11:10:49,343 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (VdsDeploy) Error during deploy dialog: java.io.IOException:
> >> Unexpected connection termination
> >> 2014-05-27 11:10:49,344 ERROR
> >> [org.ovirt.engine.core.utils.ssh.SSHDialog]
> >> (org.ovirt.thread.pool-6-thread-31) SSH error running command
> >> root@10.251.193.9:'umask 0077; MYTMP="$(mktemp -t
ovirt-XXXXXXXXXX)";
> >> trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1;
rm -fr
> >> \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr
"${MYTMP}" && mkdir
> >> "${MYTMP}" && tar --warning=no-timestamp -C
"${MYTMP}" -x &&
> >> "${MYTMP}"/setup DIALOG/dialect=str:machine
> >> DIALOG/customization=bool:True':
> >> javax.naming.TimeLimitExceededException: SSH session hard timeout host
> >> 'root(a)10.251.193.9'
> >> 2014-05-27 11:10:49,369 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-31) [26c21342] Timeout during host
> >> 10.251.193.9 install: javax.naming.TimeLimitExceededException: SSH
> >> session hard timeout host 'root(a)10.251.193.9'
> >> 2014-05-27 11:10:49,377 ERROR
> >> [org.ovirt.engine.core.bll.InstallerMessages]
> >> (org.ovirt.thread.pool-6-thread-31) [26c21342] Installation
> >> 10.251.193.9: Processing stopped due to timeout
> >> 2014-05-27 11:10:49,434 ERROR
> >> [org.ovirt.engine.core.bll.InstallVdsCommand]
> >> (org.ovirt.thread.pool-6-thread-31) [26c21342] Host installation
> >> failed for host 322cbee8-16e6-11e2-9d38-6388c61dd004,
> >> node02.blabla.gov.za.: javax.naming.TimeLimitExceededException: SSH
> >> session hard timeout host 'root(a)10.251.193.9'
> >> 2014-05-27 12:44:36,200 ERROR
> >> [org.ovirt.engine.core.utils.servlet.ServletUtils]
> >> (ajp--127.0.0.1-8702-1) Can't read file
> >> "/var/lib/ovirt-engine/reports.xml" for request
> >> "/ovirt-engine/services/reports-ui", will send a 404 error
response.
> >> 2014-05-27 13:16:21,679 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request failed with exit code 1
> >> 2014-05-27 13:16:21,680 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request script errors:
> >> Error opening Certificate ca.pem
> >> 140249235597128:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('ca.pem','r')
> >> 140249235597128:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> Error opening CA private key private/ca.pem
> >> 140630029801288:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('private/ca.pem','r')
> >> 140630029801288:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> 2014-05-27 13:16:21,684 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (VdsDeploy) Error during deploy dialog: java.lang.RuntimeException:
> >> Certificate enrollment failed
> >> 2014-05-27 13:16:21,689 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-21) [1a930dd7] Error during host
> >> 10.251.193.9 install: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 13:16:21,694 ERROR
> >> [org.ovirt.engine.core.bll.InstallerMessages]
> >> (org.ovirt.thread.pool-6-thread-21) [1a930dd7] Installation
> >> 10.251.193.9: Certificate enrollment failed
> >> 2014-05-27 13:16:21,740 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-21) [1a930dd7] Error during host
> >> 10.251.193.9 install, prefering first exception:
> >> java.lang.RuntimeException: Certificate enrollment failed
> >> 2014-05-27 13:16:21,744 ERROR
> >> [org.ovirt.engine.core.bll.InstallVdsCommand]
> >> (org.ovirt.thread.pool-6-thread-21) [1a930dd7] Host installation
> >> failed for host 322cbee8-16e6-11e2-9d38-6388c61dd004,
> >> node02.blabla.gov.za.: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 14:31:12,192 ERROR
> >> [org.ovirt.engine.core.utils.servlet.ServletUtils]
> >> (ajp--127.0.0.1-8702-2) Can't read file
> >> "/var/lib/ovirt-engine/reports.xml" for request
> >> "/ovirt-engine/services/reports-ui", will send a 404 error
response.
> >> 2014-05-27 14:32:58,669 ERROR
> >> [org.ovirt.engine.core.utils.servlet.ServletUtils]
> >> (ajp--127.0.0.1-8702-7) Can't read file
> >> "/var/lib/ovirt-engine/reports.xml" for request
> >> "/ovirt-engine/services/reports-ui", will send a 404 error
response.
> >> 2014-05-27 14:36:33,523 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request failed with exit code 1
> >> 2014-05-27 14:36:33,524 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request script errors:
> >> Error opening Certificate ca.pem
> >> 140189576382280:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('ca.pem','r')
> >> 140189576382280:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> Error opening CA private key private/ca.pem
> >> 140632037402440:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('private/ca.pem','r')
> >> 140632037402440:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> 2014-05-27 14:36:33,528 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (VdsDeploy) Error during deploy dialog: java.lang.RuntimeException:
> >> Certificate enrollment failed
> >> 2014-05-27 14:36:33,534 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-33) [5537b7c] Error during host
> >> 10.251.193.9 install: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 14:36:33,545 ERROR
> >> [org.ovirt.engine.core.bll.InstallerMessages]
> >> (org.ovirt.thread.pool-6-thread-33) [5537b7c] Installation
> >> 10.251.193.9: Certificate enrollment failed
> >> 2014-05-27 14:36:33,572 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-33) [5537b7c] Error during host
> >> 10.251.193.9 install, prefering first exception:
> >> java.lang.RuntimeException: Certificate enrollment failed
> >> 2014-05-27 14:36:33,576 ERROR
> >> [org.ovirt.engine.core.bll.InstallVdsCommand]
> >> (org.ovirt.thread.pool-6-thread-33) [5537b7c] Host installation failed
> >> for host 322cbee8-16e6-11e2-9d38-6388c61dd004, node02.blabla.gov.za.:
> >> java.lang.RuntimeException: Certificate enrollment failed
> >> 2014-05-27 14:40:26,630 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request failed with exit code 1
> >> 2014-05-27 14:40:26,631 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request script errors:
> >> Error opening Certificate ca.pem
> >> 139666318882632:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('ca.pem','r')
> >> 139666318882632:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> Error opening CA private key private/ca.pem
> >> 139701081003848:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('private/ca.pem','r')
> >> 139701081003848:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> 2014-05-27 14:40:26,633 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (VdsDeploy) Error during deploy dialog: java.lang.RuntimeException:
> >> Certificate enrollment failed
> >> 2014-05-27 14:40:26,637 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-11) [7f68b0e2] Error during host
> >> 10.251.193.9 install: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 14:40:26,639 ERROR
> >> [org.ovirt.engine.core.bll.InstallerMessages]
> >> (org.ovirt.thread.pool-6-thread-11) [7f68b0e2] Installation
> >> 10.251.193.9: Certificate enrollment failed
> >> 2014-05-27 14:40:26,709 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-11) [7f68b0e2] Error during host
> >> 10.251.193.9 install, prefering first exception:
> >> java.lang.RuntimeException: Certificate enrollment failed
> >> 2014-05-27 14:40:26,711 ERROR
> >> [org.ovirt.engine.core.bll.InstallVdsCommand]
> >> (org.ovirt.thread.pool-6-thread-11) [7f68b0e2] Host installation
> >> failed for host 322cbee8-16e6-11e2-9d38-6388c61dd004,
> >> node02.blabla.gov.za.: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 15:04:24,260 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request failed with exit code 1
> >> 2014-05-27 15:04:24,261 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request script errors:
> >> Error opening Certificate ca.pem
> >> 140668006123336:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('ca.pem','r')
> >> 140668006123336:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> Error opening CA private key private/ca.pem
> >> 140106430207816:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('private/ca.pem','r')
> >> 140106430207816:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> 2014-05-27 15:04:24,265 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (VdsDeploy) Error during deploy dialog: java.lang.RuntimeException:
> >> Certificate enrollment failed
> >> 2014-05-27 15:04:24,270 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-34) [797b7d7a] Error during host
> >> 10.251.193.9 install: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 15:04:24,277 ERROR
> >> [org.ovirt.engine.core.bll.InstallerMessages]
> >> (org.ovirt.thread.pool-6-thread-34) [797b7d7a] Installation
> >> 10.251.193.9: Certificate enrollment failed
> >> 2014-05-27 15:04:24,348 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-34) [797b7d7a] Error during host
> >> 10.251.193.9 install, prefering first exception:
> >> java.lang.RuntimeException: Certificate enrollment failed
> >> 2014-05-27 15:04:24,352 ERROR
> >> [org.ovirt.engine.core.bll.InstallVdsCommand]
> >> (org.ovirt.thread.pool-6-thread-34) [797b7d7a] Host installation
> >> failed for host 322cbee8-16e6-11e2-9d38-6388c61dd004,
> >> node02.blabla.gov.za.: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 16:48:49,075 ERROR
> >> [org.ovirt.engine.core.utils.servlet.ServletUtils]
> >> (ajp--127.0.0.1-8702-4) Can't read file
> >> "/var/lib/ovirt-engine/reports.xml" for request
> >> "/ovirt-engine/services/reports-ui", will send a 404 error
response.
> >> 2014-05-27 17:03:10,817 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request failed with exit code 1
> >> 2014-05-27 17:03:10,817 ERROR
> >> [org.ovirt.engine.core.utils.hostinstall.OpenSslCAWrapper] (VdsDeploy)
> >> Sign Certificate request script errors:
> >> Error opening Certificate ca.pem
> >> 140117678909256:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('ca.pem','r')
> >> 140117678909256:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> Error opening CA private key private/ca.pem
> >> 140049924028232:error:0200100D:system library:fopen:Permission
> >> denied:bss_file.c:398:fopen('private/ca.pem','r')
> >> 140049924028232:error:20074002:BIO routines:FILE_CTRL:system
> >> lib:bss_file.c:400:
> >> 2014-05-27 17:03:10,821 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (VdsDeploy) Error during deploy dialog: java.lang.RuntimeException:
> >> Certificate enrollment failed
> >> 2014-05-27 17:03:10,828 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-18) [2bb26823] Error during host
> >> 10.251.193.9 install: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >> 2014-05-27 17:03:10,839 ERROR
> >> [org.ovirt.engine.core.bll.InstallerMessages]
> >> (org.ovirt.thread.pool-6-thread-18) [2bb26823] Installation
> >> 10.251.193.9: Certificate enrollment failed
> >> 2014-05-27 17:03:10,891 ERROR [org.ovirt.engine.core.bll.VdsDeploy]
> >> (org.ovirt.thread.pool-6-thread-18) [2bb26823] Error during host
> >> 10.251.193.9 install, prefering first exception:
> >> java.lang.RuntimeException: Certificate enrollment failed
> >> 2014-05-27 17:03:10,895 ERROR
> >> [org.ovirt.engine.core.bll.InstallVdsCommand]
> >> (org.ovirt.thread.pool-6-thread-18) [2bb26823] Host installation
> >> failed for host d2debdfe-76e7-40cf-a7fd-78a0f50f14d4,
> >> node02.blabla.gov.za.: java.lang.RuntimeException: Certificate
> >> enrollment failed
> >>
> >> I've looked around quite a bit and can't seem to find much.
> >>
> >> Please could someone assist.
> >>
> >> Thank you.
> >>
> >> Regards,
> >>
> >> Neil Wilson.
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org
> >>
http://lists.ovirt.org/mailman/listinfo/users
> >>
>