problems installing standard Linux as nodes in 4.4
Hello, due to missing megaraid_sas kernel module in RH EL 8 and CentOS 8, I'm deploying a new oVirt host using CentOS 8 and elrepo kernel driver and not ovirt node ng. Based on installation guide: - install CentOS 8.2 ("Server" chosen as base environment) - yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm - yum install cockpit-ovirt-dashboard - yum update - reboot Try to add host from engine web admin gui, I get: Host ov200 installation failed. Failed to execute Ansible host-deploy role: Failed to execute call to start playbook. . Please check logs for more details: /var/log/ovirt-engine/ansible-runner-service.log. Inside the log file above on engine: 2020-10-08 11:58:43,389 - runner_service.controllers.hosts - DEBUG - Request received, content-type :None 2020-10-08 11:58:43,390 - runner_service.controllers.hosts - INFO - 127.0.0.1 - GET /api/v1/hosts/ov200 2020-10-08 11:58:43,398 - runner_service.controllers.playbooks - DEBUG - Request received, content-type :application/json; charset=UTF-8 2020-10-08 11:58:43,398 - runner_service.controllers.playbooks - INFO - 127.0.0.1 - POST /api/v1/playbooks/ovirt-host-deploy.yml Do I have to enable any module or pre-install anything else before adding it? BTW: on host [root@ov200 ~]# rpm -q ansible ansible-2.9.13-2.el8.noarch [root@ov200 ~]# Thanks, Gianluca
Hi Gianluca, Please execute the following command on your engine, save the output into a file and attach it: sudo journalctl -u ansible-runner-service Dana On Thu, Oct 8, 2020 at 1:13 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, due to missing megaraid_sas kernel module in RH EL 8 and CentOS 8, I'm deploying a new oVirt host using CentOS 8 and elrepo kernel driver and not ovirt node ng.
Based on installation guide:
- install CentOS 8.2 ("Server" chosen as base environment) - yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm - yum install cockpit-ovirt-dashboard - yum update - reboot
Try to add host from engine web admin gui, I get: Host ov200 installation failed. Failed to execute Ansible host-deploy role: Failed to execute call to start playbook. . Please check logs for more details: /var/log/ovirt-engine/ansible-runner-service.log.
Inside the log file above on engine: 2020-10-08 11:58:43,389 - runner_service.controllers.hosts - DEBUG - Request received, content-type :None 2020-10-08 11:58:43,390 - runner_service.controllers.hosts - INFO - 127.0.0.1 - GET /api/v1/hosts/ov200 2020-10-08 11:58:43,398 - runner_service.controllers.playbooks - DEBUG - Request received, content-type :application/json; charset=UTF-8 2020-10-08 11:58:43,398 - runner_service.controllers.playbooks - INFO - 127.0.0.1 - POST /api/v1/playbooks/ovirt-host-deploy.yml
Do I have to enable any module or pre-install anything else before adding it?
BTW: on host
[root@ov200 ~]# rpm -q ansible ansible-2.9.13-2.el8.noarch [root@ov200 ~]#
Thanks, Gianluca
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FT4S5MRJGJJNMG...
And also please attach the content of the file found at: /etc/ansible-runner-service/config.yaml On Thu, Oct 8, 2020 at 5:55 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi Gianluca, Please execute the following command on your engine, save the output into a file and attach it: sudo journalctl -u ansible-runner-service Dana
On Thu, Oct 8, 2020 at 1:13 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, due to missing megaraid_sas kernel module in RH EL 8 and CentOS 8, I'm deploying a new oVirt host using CentOS 8 and elrepo kernel driver and not ovirt node ng.
Based on installation guide:
- install CentOS 8.2 ("Server" chosen as base environment) - yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm - yum install cockpit-ovirt-dashboard - yum update - reboot
Try to add host from engine web admin gui, I get: Host ov200 installation failed. Failed to execute Ansible host-deploy role: Failed to execute call to start playbook. . Please check logs for more details: /var/log/ovirt-engine/ansible-runner-service.log.
Inside the log file above on engine: 2020-10-08 11:58:43,389 - runner_service.controllers.hosts - DEBUG - Request received, content-type :None 2020-10-08 11:58:43,390 - runner_service.controllers.hosts - INFO - 127.0.0.1 - GET /api/v1/hosts/ov200 2020-10-08 11:58:43,398 - runner_service.controllers.playbooks - DEBUG - Request received, content-type :application/json; charset=UTF-8 2020-10-08 11:58:43,398 - runner_service.controllers.playbooks - INFO - 127.0.0.1 - POST /api/v1/playbooks/ovirt-host-deploy.yml
Do I have to enable any module or pre-install anything else before adding it?
BTW: on host
[root@ov200 ~]# rpm -q ansible ansible-2.9.13-2.el8.noarch [root@ov200 ~]#
Thanks, Gianluca
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FT4S5MRJGJJNMG...
On Thu, Oct 8, 2020 at 4:59 PM Dana Elfassy <delfassy@redhat.com> wrote:
And also please attach the content of the file found at: /etc/ansible-runner-service/config.yaml
On Thu, Oct 8, 2020 at 5:55 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi Gianluca, Please execute the following command on your engine, save the output into a file and attach it: sudo journalctl -u ansible-runner-service Dana
Thanks for answering, Dana.
[root@ovmgr1 ansible-runner-service]# sudo journalctl -u ansible-runner-service -- Logs begin at Tue 2020-10-06 11:12:46 CEST, end at Thu 2020-10-08 17:02:25 CEST. -- -- No entries -- [root@ovmgr1 ansible-runner-service]# [root@ovmgr1 ansible-runner-service]# cat /etc/ansible-runner-service/config.yaml version: 1 playbooks_root_dir: '/usr/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root log_path: '/var/log/ovirt-engine' [root@ovmgr1 ansible-runner-service]# I noticed that both on engine and on host the "ansible-runner" package is not installed. Is it correct and only ansible-runner-service package to be installed only on the engine? Also, does the "service" in the name imply that I should have any systemd or other kind of related service on engine? Finally, I have to use a proxy for dnf/yum. To be able to run "engine-setup" on engine I had to set http_proxy and https_proxy eng variables inside the shell session, because it seems that engine-setup was not able to leverage the global configuration. Could it be something similar due to the host having to use a proxy too (that I already setup in /etc/dnf/dnf.conf)? Just a guess. Gianluca
On Thu, Oct 8, 2020 at 5:08 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 8, 2020 at 4:59 PM Dana Elfassy <delfassy@redhat.com> wrote:
And also please attach the content of the file found at: /etc/ansible-runner-service/config.yaml
On Thu, Oct 8, 2020 at 5:55 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi Gianluca, Please execute the following command on your engine, save the output into a file and attach it: sudo journalctl -u ansible-runner-service Dana
Thanks for answering, Dana.
[root@ovmgr1 ansible-runner-service]# sudo journalctl -u ansible-runner-service -- Logs begin at Tue 2020-10-06 11:12:46 CEST, end at Thu 2020-10-08 17:02:25 CEST. -- -- No entries -- [root@ovmgr1 ansible-runner-service]#
[root@ovmgr1 ansible-runner-service]# cat /etc/ansible-runner-service/config.yaml
version: 1 playbooks_root_dir: '/usr/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root log_path: '/var/log/ovirt-engine' [root@ovmgr1 ansible-runner-service]#
I noticed that both on engine and on host the "ansible-runner" package is not installed. Is it correct and only ansible-runner-service package to be installed only on the engine? Also, does the "service" in the name imply that I should have any systemd or other kind of related service on engine? Finally, I have to use a proxy for dnf/yum. To be able to run "engine-setup" on engine I had to set http_proxy and https_proxy eng variables inside the shell session, because it seems that engine-setup was not able to leverage the global configuration. Could it be something similar due to the host having to use a proxy too (that I already setup in /etc/dnf/dnf.conf)? Just a guess.
Gianluca
Also, the host already existed in 4.3. I upgraded the standalone engine from 4.3.10 to 4.4.2 following the guide. Now to update my hosts I put a host into maintenance, removed the host from the gui, reinstalled the server in CentOS 8.2 with same network parameters, and then add new host with the same name/hostname as before. Could it be a problem to reuse the host? Gianluca
On Thu, Oct 8, 2020 at 5:13 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 8, 2020 at 5:08 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 8, 2020 at 4:59 PM Dana Elfassy <delfassy@redhat.com> wrote:
And also please attach the content of the file found at: /etc/ansible-runner-service/config.yaml
On Thu, Oct 8, 2020 at 5:55 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi Gianluca, Please execute the following command on your engine, save the output into a file and attach it: sudo journalctl -u ansible-runner-service Dana
Thanks for answering, Dana.
[root@ovmgr1 ansible-runner-service]# sudo journalctl -u ansible-runner-service -- Logs begin at Tue 2020-10-06 11:12:46 CEST, end at Thu 2020-10-08 17:02:25 CEST. -- -- No entries -- [root@ovmgr1 ansible-runner-service]#
[root@ovmgr1 ansible-runner-service]# cat /etc/ansible-runner-service/config.yaml
version: 1 playbooks_root_dir: '/usr/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root log_path: '/var/log/ovirt-engine' [root@ovmgr1 ansible-runner-service]#
I noticed that both on engine and on host the "ansible-runner" package is not installed. Is it correct and only ansible-runner-service package to be installed only on the engine? Also, does the "service" in the name imply that I should have any systemd or other kind of related service on engine? Finally, I have to use a proxy for dnf/yum. To be able to run "engine-setup" on engine I had to set http_proxy and https_proxy eng variables inside the shell session, because it seems that engine-setup was not able to leverage the global configuration. Could it be something similar due to the host having to use a proxy too (that I already setup in /etc/dnf/dnf.conf)? Just a guess.
Gianluca
Also, the host already existed in 4.3. I upgraded the standalone engine from 4.3.10 to 4.4.2 following the guide. Now to update my hosts I put a host into maintenance, removed the host from the gui, reinstalled the server in CentOS 8.2 with same network parameters, and then add new host with the same name/hostname as before. Could it be a problem to reuse the host?
Gianluca
Any other thing to check to be able to provision a node in 4.4.2 using plain CentOS 8.2 host? Thanks, Gianluca
Hi Gianluca, could you please check selinux context of /var/log/ovirt-engine/ansible-runner-service.log to see if you are not affected by https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 ? Thanks, Martin On Fri, Oct 9, 2020 at 4:45 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 8, 2020 at 5:13 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 8, 2020 at 5:08 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 8, 2020 at 4:59 PM Dana Elfassy <delfassy@redhat.com> wrote:
And also please attach the content of the file found at: /etc/ansible-runner-service/config.yaml
On Thu, Oct 8, 2020 at 5:55 PM Dana Elfassy <delfassy@redhat.com> wrote:
Hi Gianluca, Please execute the following command on your engine, save the output into a file and attach it: sudo journalctl -u ansible-runner-service Dana
Thanks for answering, Dana.
[root@ovmgr1 ansible-runner-service]# sudo journalctl -u ansible-runner-service -- Logs begin at Tue 2020-10-06 11:12:46 CEST, end at Thu 2020-10-08 17:02:25 CEST. -- -- No entries -- [root@ovmgr1 ansible-runner-service]#
[root@ovmgr1 ansible-runner-service]# cat /etc/ansible-runner-service/config.yaml
version: 1 playbooks_root_dir: '/usr/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root log_path: '/var/log/ovirt-engine' [root@ovmgr1 ansible-runner-service]#
I noticed that both on engine and on host the "ansible-runner" package is not installed. Is it correct and only ansible-runner-service package to be installed only on the engine? Also, does the "service" in the name imply that I should have any systemd or other kind of related service on engine? Finally, I have to use a proxy for dnf/yum. To be able to run "engine-setup" on engine I had to set http_proxy and https_proxy eng variables inside the shell session, because it seems that engine-setup was not able to leverage the global configuration. Could it be something similar due to the host having to use a proxy too (that I already setup in /etc/dnf/dnf.conf)? Just a guess.
Gianluca
Also, the host already existed in 4.3. I upgraded the standalone engine from 4.3.10 to 4.4.2 following the guide. Now to update my hosts I put a host into maintenance, removed the host from the gui, reinstalled the server in CentOS 8.2 with same network parameters, and then add new host with the same name/hostname as before. Could it be a problem to reuse the host?
Gianluca
Any other thing to check to be able to provision a node in 4.4.2 using plain CentOS 8.2 host? Thanks, Gianluca _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AJN3ENCAXNCTGW...
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Oct 9, 2020 at 4:58 PM Martin Perina <mperina@redhat.com> wrote:
Hi Gianluca,
could you please check selinux context of /var/log/ovirt-engine/ansible-runner-service.log to see if you are not affected by https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 ?
Thanks, Martin
Thanks for answering. It seems ok. On the engine: [root@ovmgr1 ~]# ls -Z /var/log/ovirt-engine/ansible-runner-service.log system_u:object_r:httpd_log_t:s0 /var/log/ovirt-engine/ansible-runner-service.log [root@ovmgr1 ~]# Gianluca
On Fri, Oct 9, 2020 at 5:54 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Oct 9, 2020 at 4:58 PM Martin Perina <mperina@redhat.com> wrote:
Hi Gianluca,
could you please check selinux context of /var/log/ovirt-engine/ansible-runner-service.log to see if you are not affected by https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 ?
Thanks, Martin
Thanks for answering. It seems ok. On the engine: [root@ovmgr1 ~]# ls -Z /var/log/ovirt-engine/ansible-runner-service.log system_u:object_r:httpd_log_t:s0 /var/log/ovirt-engine/ansible-runner-service.log [root@ovmgr1 ~]#
Gianluca
OK, so could you please apply the workaround mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 to resolve the issue until oVirt 4.4.3 is released? -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Oct 9, 2020 at 6:29 PM Martin Perina <mperina@redhat.com> wrote:
On Fri, Oct 9, 2020 at 5:54 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Oct 9, 2020 at 4:58 PM Martin Perina <mperina@redhat.com> wrote:
Hi Gianluca,
could you please check selinux context of /var/log/ovirt-engine/ansible-runner-service.log to see if you are not affected by https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 ?
Thanks, Martin
Thanks for answering. It seems ok. On the engine: [root@ovmgr1 ~]# ls -Z /var/log/ovirt-engine/ansible-runner-service.log system_u:object_r:httpd_log_t:s0 /var/log/ovirt-engine/ansible-runner-service.log [root@ovmgr1 ~]#
Gianluca
OK, so could you please apply the workaround mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 to resolve the issue until oVirt 4.4.3 is released?
Sorry, but isn't it already ok? The SELinux security context for the file is already httpd_log_t, so I don't have to apply anything. I also applied the more brutal workaround described in https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c4 without any effect, so I'm not in this bugzilla context. Do I have to apply also for the directory /var/log/ovirt-engine itself, that currently has a context of var_log_t? I don't think so... Gianluca
On Fri, Oct 9, 2020 at 6:47 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Oct 9, 2020 at 6:29 PM Martin Perina <mperina@redhat.com> wrote:
On Fri, Oct 9, 2020 at 5:54 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Oct 9, 2020 at 4:58 PM Martin Perina <mperina@redhat.com> wrote:
Hi Gianluca,
could you please check selinux context of /var/log/ovirt-engine/ansible-runner-service.log to see if you are not affected by https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 ?
Thanks, Martin
Thanks for answering. It seems ok. On the engine: [root@ovmgr1 ~]# ls -Z /var/log/ovirt-engine/ansible-runner-service.log system_u:object_r:httpd_log_t:s0 /var/log/ovirt-engine/ansible-runner-service.log [root@ovmgr1 ~]#
Gianluca
OK, so could you please apply the workaround mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c5 to resolve the issue until oVirt 4.4.3 is released?
Sorry, but isn't it already ok? The SELinux security context for the file is already httpd_log_t, so I don't have to apply anything. I also applied the more brutal workaround described in https://bugzilla.redhat.com/show_bug.cgi?id=1880171#c4 without any effect, so I'm not in this bugzilla context. Do I have to apply also for the directory /var/log/ovirt-engine itself, that currently has a context of var_log_t? I don't think so...
Ahh, sorry, I've misunderstood your reply, I thought you replied you are affected. Could you please share with us all logs from engine gathered by logcollector? We will try to find out any clue what's wrong in your env ... Thanks, Martin
Gianluca
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Oct 9, 2020 at 7:12 PM Martin Perina <mperina@redhat.com> wrote:
Could you please share with us all logs from engine gathered by logcollector? We will try to find out any clue what's wrong in your env ...
Thanks, Martin
I will try to collect. In the mean time I've found that SSH could be in some way involved When I add the host and get the immediate failure and apparently nothing happens at all, I see these two lines in /var/log/ovirt-engine/server.log 2020-10-09 18:15:09,369+02 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[7cb54873]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200 /10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported 2020-10-09 18:15:09,699+02 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[2cbceeab]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200 /10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported could it be that the ssh client embedded is not able to connect to the CentOS 8.2 for some reason? On host at the moment when I try to add it I see again two sessions opened and immediately closed (tried several times), eg in the timeframe above I have: Oct 9 18:15:09 ov200 systemd-logind[1237]: New session 41 of user root. Oct 9 18:15:09 ov200 systemd[1]: Started Session 41 of user root. Oct 9 18:15:09 ov200 systemd-logind[1237]: Session 41 logged out. Waiting for processes to exit. Oct 9 18:15:09 ov200 systemd-logind[1237]: Removed session 41. Oct 9 18:15:09 ov200 systemd-logind[1237]: New session 42 of user root. Oct 9 18:15:09 ov200 systemd[1]: Started Session 42 of user root. Oct 9 18:15:09 ov200 systemd-logind[1237]: Session 42 logged out. Waiting for processes to exit. Oct 9 18:15:09 ov200 systemd-logind[1237]: Removed session 42. anyway at sshd service level it seems it is ok om the host: journalctl -u sshd.service has Oct 09 18:15:09 ov200 sshd[13379]: Accepted password for root from 10.4.192.43 port 46008 ssh2 Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session opened for user root by (uid=0) Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session closed for user root Oct 09 18:15:09 ov200 sshd[13398]: Accepted password for root from 10.4.192.43 port 46014 ssh2 Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session opened for user root by (uid=0) Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session closed for user root On the host I have not customized anything ssh related: [root@ov200 ssh]# ps -ef|grep sshd root 1274 1 0 Oct08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs= hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com, umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com ,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com ,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,rsa-sha2-512, rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521, ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519, ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com ,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com ,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,rsa-sha2-512, rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521, ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519, ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=rsa-sha2-256,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,rsa-sha2-512,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa and in sshd_config HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_ed25519_key Can I replicate the command that the engine would run on host through ssh? Gianluca
I guess you tried to ssh to the HostedEngine and then ssh to the host , right ? Best Regards, Strahil Nikolov В събота, 10 октомври 2020 г., 02:28:35 Гринуич+3, Gianluca Cecchi <gianluca.cecchi@gmail.com> написа: On Fri, Oct 9, 2020 at 7:12 PM Martin Perina <mperina@redhat.com> wrote:
Could you please share with us all logs from engine gathered by logcollector? We will try to find out any clue what's wrong in your env ...
Thanks, Martin
I will try to collect. In the mean time I've found that SSH could be in some way involved When I add the host and get the immediate failure and apparently nothing happens at all, I see these two lines in /var/log/ovirt-engine/server.log 2020-10-09 18:15:09,369+02 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[7cb54873]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200/10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported 2020-10-09 18:15:09,699+02 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[2cbceeab]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200/10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported could it be that the ssh client embedded is not able to connect to the CentOS 8.2 for some reason? On host at the moment when I try to add it I see again two sessions opened and immediately closed (tried several times), eg in the timeframe above I have: Oct 9 18:15:09 ov200 systemd-logind[1237]: New session 41 of user root. Oct 9 18:15:09 ov200 systemd[1]: Started Session 41 of user root. Oct 9 18:15:09 ov200 systemd-logind[1237]: Session 41 logged out. Waiting for processes to exit. Oct 9 18:15:09 ov200 systemd-logind[1237]: Removed session 41. Oct 9 18:15:09 ov200 systemd-logind[1237]: New session 42 of user root. Oct 9 18:15:09 ov200 systemd[1]: Started Session 42 of user root. Oct 9 18:15:09 ov200 systemd-logind[1237]: Session 42 logged out. Waiting for processes to exit. Oct 9 18:15:09 ov200 systemd-logind[1237]: Removed session 42. anyway at sshd service level it seems it is ok om the host: journalctl -u sshd.service has Oct 09 18:15:09 ov200 sshd[13379]: Accepted password for root from 10.4.192.43 port 46008 ssh2 Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session opened for user root by (uid=0) Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session closed for user root Oct 09 18:15:09 ov200 sshd[13398]: Accepted password for root from 10.4.192.43 port 46014 ssh2 Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session opened for user root by (uid=0) Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session closed for user root On the host I have not customized anything ssh related: [root@ov200 ssh]# ps -ef|grep sshd root 1274 1 0 Oct08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=rsa-sha2-256,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,rsa-sha2-512,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa and in sshd_config HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_ed25519_key Can I replicate the command that the engine would run on host through ssh? Gianluca _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QNZP6PA6XWFMXK...
On Sat, 10 Oct 2020, 01:24 Gianluca Cecchi, <gianluca.cecchi@gmail.com> wrote:
On Fri, Oct 9, 2020 at 7:12 PM Martin Perina <mperina@redhat.com> wrote:
Could you please share with us all logs from engine gathered by logcollector? We will try to find out any clue what's wrong in your env ...
Thanks, Martin
I will try to collect. In the mean time I've found that SSH could be in some way involved
When I add the host and get the immediate failure and apparently nothing happens at all, I see these two lines in /var/log/ovirt-engine/server.log
2020-10-09 18:15:09,369+02 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[7cb54873]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200 /10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported 2020-10-09 18:15:09,699+02 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[2cbceeab]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200 /10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported
This harmless, AFAIK EdDSA is not supported by default in OpenJDK 11 and engine uses only ssh-rsa and ssh-rsa2 anyway
could it be that the ssh client embedded is not able to connect to the CentOS 8.2 for some reason?
If that's the case we should see an error either in engine.log or ansible-runner-service.log
On host at the moment when I try to add it I see again two sessions opened and immediately closed (tried several times), eg in the timeframe above I have:
Oct 9 18:15:09 ov200 systemd-logind[1237]: New session 41 of user root. Oct 9 18:15:09 ov200 systemd[1]: Started Session 41 of user root. Oct 9 18:15:09 ov200 systemd-logind[1237]: Session 41 logged out. Waiting for processes to exit. Oct 9 18:15:09 ov200 systemd-logind[1237]: Removed session 41. Oct 9 18:15:09 ov200 systemd-logind[1237]: New session 42 of user root. Oct 9 18:15:09 ov200 systemd[1]: Started Session 42 of user root. Oct 9 18:15:09 ov200 systemd-logind[1237]: Session 42 logged out. Waiting for processes to exit. Oct 9 18:15:09 ov200 systemd-logind[1237]: Removed session 42.
anyway at sshd service level it seems it is ok om the host:
journalctl -u sshd.service has
Oct 09 18:15:09 ov200 sshd[13379]: Accepted password for root from 10.4.192.43 port 46008 ssh2 Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session opened for user root by (uid=0) Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session closed for user root Oct 09 18:15:09 ov200 sshd[13398]: Accepted password for root from 10.4.192.43 port 46014 ssh2 Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session opened for user root by (uid=0) Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session closed for user root
On the host I have not customized anything ssh related:
[root@ov200 ssh]# ps -ef|grep sshd root 1274 1 0 Oct08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com, umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com ,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com ,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com ,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521, ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519, ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com ,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com ,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com ,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521, ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519, ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=rsa-sha2-256,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,rsa-sha2-512,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa
and in sshd_config
HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_ed25519_key
That looks good
Can I replicate the command that the engine would run on host through ssh?
I don't think so there is an easy way to do it Let's see what else we can get from the logs... Martin
Gianluca
On Sat, Oct 10, 2020 at 10:13 AM Martin Perina <mperina@redhat.com> wrote: [snip]
Can I replicate the command that the engine would run on host through ssh?
I don't think so there is an easy way to do it Let's see what else we can get from the logs...
Martin
Hi, I've run on engine the command ovirt-log-collector --no-hypervisors but potentially there is much sensitive information (like the dump of the database). Is there any particular file you are more interested in that archive I can share? BTW: can I put engine in debug for the time I'm trying to add the host so that we can see if more messages are shown? In that case how can I do? Another information I have noticed is that when the new host command from web admin GUI suddenly fails, anyway the ov200 host is now present in the host list, with the down icon and "Install failed" info. If I click on it and go in General subtab, in the section "Action Items" I see 3 items with exclamation mark in front of them: 1) Power Management is not configured for this Host. Enable Power Management --> OK, I skipped it 2) Host has no default route. ---> I don't know why it says this. [root@ov200 log]# ip route show default via 10.4.192.254 dev bond0.68 proto static metric 400 10.4.192.0/24 dev bond0.68 proto kernel scope link src 10.4.192.32 metric 400 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [root@ov200 log]# On the still in CentOS 7 active host I have: [root@ov300 ~]# ip route show default via 10.4.192.254 dev ovirtmgmntZ2Z3 10.4.187.0/24 dev p1p2.187 proto kernel scope link src 10.4.187.100 10.4.192.0/24 dev ovirtmgmntZ2Z3 proto kernel scope link src 10.4.192.33 10.10.100.0/24 dev p1p2 proto kernel scope link src 10.10.100.88 10.10.100.0/24 dev p1p1.100 proto kernel scope link src 10.10.100.87 [root@ov300 ~]# [root@ov300 ~]# brctl show ovirtmgmntZ2Z3 bridge name bridge id STP enabled interfaces ovirtmgmntZ2Z3 8000.1803730ba369 no bond0.68 [root@ov300 ~]# Could it be the fact that for historical reasons my mgmt network has not the name ovirtmgmt but ovirtmgmntZ2Z3 that confuses the installer that expects ovirtmgmt to setup? And erroneously reports the no default route message? 3) The host CPU does not match the Cluster CPU Type and is running in a degraded mode. It is missing the following CPU flags: vmx, ssbd, nx, model_Westmere, aes, spec_ctrl. Please update the host CPU microcode or change the Cluster CPU Type. The cluster is set as "Intel Westmere IBRS SSBD Family". all the hosts are the same hw Dell PE M610, with same processor Host installed in CentOS 8: [root@ov200 log]# cat /proc/cpuinfo | grep "model name" | sort -u model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz [root@ov200 log]# Host still in CentOS 7: [root@ov300 ~]# cat /proc/cpuinfo | grep "model name" | sort -u model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz [root@ov300 ~]# If I compare the cpu flags inside the OS I see: CentOS 8: [root@ov200 log]# cat /proc/cpuinfo | grep flags | sort -u flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat flush_l1d [root@ov200 log]# CentOS 7: [root@ov300 ~]# cat /proc/cpuinfo | grep flags | sort -u flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1d [root@ov300 ~]# When still in CentOS 7, ov200 had the same flags as ov300 ov200 has this more now: cpuid pti ov200 has these less now: eagerfpu spec_ctrl intel_stibp Gianluca
On Tue, Oct 13, 2020 at 12:06 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Sat, Oct 10, 2020 at 10:13 AM Martin Perina <mperina@redhat.com> wrote:
[snip]
Can I replicate the command that the engine would run on host through ssh?
I don't think so there is an easy way to do it Let's see what else we can get from the logs...
Martin
Hi, I've run on engine the command ovirt-log-collector --no-hypervisors but potentially there is much sensitive information (like the dump of the database).
Is there any particular file you are more interested in that archive I can share?
BTW: can I put engine in debug for the time I'm trying to add the host so that we can see if more messages are shown? In that case how can I do?
Another information I have noticed is that when the new host command from web admin GUI suddenly fails, anyway the ov200 host is now present in the host list, with the down icon and "Install failed" info. If I click on it and go in General subtab, in the section "Action Items" I see 3 items with exclamation mark in front of them:
1) Power Management is not configured for this Host. Enable Power Management --> OK, I skipped it
2) Host has no default route. ---> I don't know why it says this.
[root@ov200 log]# ip route show default via 10.4.192.254 dev bond0.68 proto static metric 400 10.4.192.0/24 dev bond0.68 proto kernel scope link src 10.4.192.32 metric 400 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [root@ov200 log]#
On the still in CentOS 7 active host I have:
[root@ov300 ~]# ip route show default via 10.4.192.254 dev ovirtmgmntZ2Z3 10.4.187.0/24 dev p1p2.187 proto kernel scope link src 10.4.187.100 10.4.192.0/24 dev ovirtmgmntZ2Z3 proto kernel scope link src 10.4.192.33 10.10.100.0/24 dev p1p2 proto kernel scope link src 10.10.100.88 10.10.100.0/24 dev p1p1.100 proto kernel scope link src 10.10.100.87 [root@ov300 ~]#
[root@ov300 ~]# brctl show ovirtmgmntZ2Z3 bridge name bridge id STP enabled interfaces ovirtmgmntZ2Z3 8000.1803730ba369 no bond0.68 [root@ov300 ~]#
Could it be the fact that for historical reasons my mgmt network has not the name ovirtmgmt but ovirtmgmntZ2Z3 that confuses the installer that expects ovirtmgmt to setup? And erroneously reports the no default route message?
3) The host CPU does not match the Cluster CPU Type and is running in a degraded mode. It is missing the following CPU flags: vmx, ssbd, nx, model_Westmere, aes, spec_ctrl. Please update the host CPU microcode or change the Cluster CPU Type.
The cluster is set as "Intel Westmere IBRS SSBD Family". all the hosts are the same hw Dell PE M610, with same processor
Host installed in CentOS 8: [root@ov200 log]# cat /proc/cpuinfo | grep "model name" | sort -u model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz [root@ov200 log]#
Host still in CentOS 7: [root@ov300 ~]# cat /proc/cpuinfo | grep "model name" | sort -u model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz [root@ov300 ~]#
If I compare the cpu flags inside the OS I see:
CentOS 8: [root@ov200 log]# cat /proc/cpuinfo | grep flags | sort -u flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat flush_l1d [root@ov200 log]#
CentOS 7: [root@ov300 ~]# cat /proc/cpuinfo | grep flags | sort -u flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1d [root@ov300 ~]#
When still in CentOS 7, ov200 had the same flags as ov300 ov200 has this more now: cpuid pti
ov200 has these less now: eagerfpu spec_ctrl intel_stibp
Gianluca
Any feedback on my latest comments? In the meantime here: https://drive.google.com/file/d/1iN37znRtCo2vgyGTH_ymLhBJfs-2pWDr/view?usp=s... you can find inside the sosreport in tar.gz format, where I have modified some file names and context in respect of hostnames. The only file I have not put inside is the dump of the database, but I can run any query you like in case. Gianluca
On Thu, Oct 15, 2020 at 10:41 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Any feedback on my latest comments? In the meantime here:
https://drive.google.com/file/d/1iN37znRtCo2vgyGTH_ymLhBJfs-2pWDr/view?usp=s... you can find inside the sosreport in tar.gz format, where I have modified some file names and context in respect of hostnames. The only file I have not put inside is the dump of the database, but I can run any query you like in case.
Gianluca
I have also tried to put debug into the engine. Method used base on this link: https://www.ovirt.org/develop/developer-guide/engine/engine-development-envi... and I used engine.core as the package [root@ovmgr1 ~]# diff ovirt-engine.xml.in ovirt-engine.xml.in.debug 118c118 < <level name="ALL"/> ---
<level name="DEBUG"/>
197a198,200
<logger category="org.ovirt.engine.core"> <level name="DEBUG"/> </logger>
[root@ovmgr1 ~]# When the install fails I get this in engine.log now: 2020-10-15 12:16:15,394+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-31) [c439aded-ade3-4474-a5f1-2f074ed5d920] EVENT_ID: VDS_ANSIBLE_INSTALL_STARTED(560), Ansible host-deploy playbook execution has started on host ov200. 2020-10-15 12:16:15,412+02 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engine-Thread-31) [c439aded-ade3-4474-a5f1-2f074ed5d920] Exception: Failed to execute call to start playbook. 2020-10-15 12:16:15,412+02 DEBUG [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engine-Thread-31) [c439aded-ade3-4474-a5f1-2f074ed5d920] Exception: : org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerCallException: Failed to execute call to start playbook. at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerHTTPClient.runPlaybook(AnsibleRunnerHTTPClient.java:153) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:113) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:78) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.runAnsibleHostDeployPlaybook(InstallVdsInternalCommand.java:281) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:145) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1169) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1327) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2003) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1387) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.executeValidatedCommand(PrevalidatingMultipleActionsRunner.java:204) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.runCommands(PrevalidatingMultipleActionsRunner.java:176) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner.lambda$invokeCommands$3(PrevalidatingMultipleActionsRunner.java:182) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) 2020-10-15 12:16:15,412+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-31) [c439aded-ade3-4474-a5f1-2f074ed5d920] Host installation failed for host '79da834f-d03a-4abc-b89e-8ad0186c173c', 'ov200': Failed to execute Ansible host-deploy role: Failed to execute call to start playbook. . Please check logs for more details: /var/log/ovirt-engine/ansible-runner-service.log Any insight? Gianluca
On Thu, Oct 15, 2020 at 12:25 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 15, 2020 at 10:41 AM Gianluca Cecchi < gianluca.cecchi@gmail.com> wrote:
Any feedback on my latest comments? In the meantime here:
https://drive.google.com/file/d/1iN37znRtCo2vgyGTH_ymLhBJfs-2pWDr/view?usp=s... you can find inside the sosreport in tar.gz format, where I have modified some file names and context in respect of hostnames. The only file I have not put inside is the dump of the database, but I can run any query you like in case.
Gianluca
I have also tried to put debug into the engine.
So after huge debugging work with Dana Elfassy and Martin Necas (thank you very much to both!) and coordination of Sandro we found the culprit! Inside firewall custom rules of my engine I had this (note the double quotes for the comment about Nagios): [root@ovmgr1 ovirt-engine]# engine-config -g IPTablesConfigSiteCustom IPTablesConfigSiteCustom: -A INPUT -p tcp --dport 5666 -s 10.4.5.99/32 -m comment --comment "Nagios NRPE daemon" -j ACCEPT version: general [root@ovmgr1 ovirt-engine]# So those double quotes caused a wrong formatted json block that ansible-runner-service was not able to manage in the http post phase After changing with single quotes, with this command: engine-config -s IPTablesConfigSiteCustom="-A INPUT -p tcp --dport 5666 -s 10.4.5.99/32 -m comment --comment 'Nagios NRPE daemon' -j ACCEPT" and restarting the engine so that now I have [root@ovmgr1 ovirt-engine]# engine-config -g IPTablesConfigSiteCustom IPTablesConfigSiteCustom: -A INPUT -p tcp --dport 5666 -s 10.4.5.99/32 -m comment --comment 'Nagios NRPE daemon' -j ACCEPT version: general [root@ovmgr1 ovirt-engine]# I was able to add the CentOS 8.2 host. So mind if you have the double quotes in any engine-config key before upgrading from 4.3 to 4.4. What a nasty thing to detect... Thanks again guys for your help Gianluca
Hi Gianluca, happy to hear that your issue was fixed! Just please be aware that iptables support for hosts has been deprecated and it's completely unsupported for cluster levels 4.4 and up. So unless you switch your cluster to firewalld, you will not be able to upgrade your cluster to 4.4 version. You can take a look at documentation how to prepare custom firewall rules for firewalld: https://www.ovirt.org/documentation/administration_guide/#Configuring_Host_F... Regards, Martin On Mon, Oct 26, 2020 at 7:22 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 15, 2020 at 12:25 PM Gianluca Cecchi < gianluca.cecchi@gmail.com> wrote:
On Thu, Oct 15, 2020 at 10:41 AM Gianluca Cecchi < gianluca.cecchi@gmail.com> wrote:
Any feedback on my latest comments? In the meantime here:
https://drive.google.com/file/d/1iN37znRtCo2vgyGTH_ymLhBJfs-2pWDr/view?usp=s... you can find inside the sosreport in tar.gz format, where I have modified some file names and context in respect of hostnames. The only file I have not put inside is the dump of the database, but I can run any query you like in case.
Gianluca
I have also tried to put debug into the engine.
So after huge debugging work with Dana Elfassy and Martin Necas (thank you very much to both!) and coordination of Sandro we found the culprit!
Inside firewall custom rules of my engine I had this (note the double quotes for the comment about Nagios):
[root@ovmgr1 ovirt-engine]# engine-config -g IPTablesConfigSiteCustom IPTablesConfigSiteCustom: -A INPUT -p tcp --dport 5666 -s 10.4.5.99/32 -m comment --comment "Nagios NRPE daemon" -j ACCEPT version: general [root@ovmgr1 ovirt-engine]#
So those double quotes caused a wrong formatted json block that ansible-runner-service was not able to manage in the http post phase
After changing with single quotes, with this command:
engine-config -s IPTablesConfigSiteCustom="-A INPUT -p tcp --dport 5666 -s 10.4.5.99/32 -m comment --comment 'Nagios NRPE daemon' -j ACCEPT"
and restarting the engine so that now I have
[root@ovmgr1 ovirt-engine]# engine-config -g IPTablesConfigSiteCustom IPTablesConfigSiteCustom: -A INPUT -p tcp --dport 5666 -s 10.4.5.99/32 -m comment --comment 'Nagios NRPE daemon' -j ACCEPT version: general [root@ovmgr1 ovirt-engine]#
I was able to add the CentOS 8.2 host. So mind if you have the double quotes in any engine-config key before upgrading from 4.3 to 4.4.
What a nasty thing to detect... Thanks again guys for your help
Gianluca
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
participants (4)
-
Dana Elfassy -
Gianluca Cecchi -
Martin Perina -
Strahil Nikolov