11:38 a.m.
I'm struggling installing the metrics store VMs. It appears that etcd image fails to
download or build a template. Thank you ahead of time for your assistance.
2019-05-28 20:07:12,650 p=22689 u=root | Tuesday 28 May 2019 20:07:12 -0400
(0:00:00.097) 0:04:54.774 ***********
2019-05-28 20:07:12,687 p=22689 u=root | fatal: [master0.ent.co.ventura.ca.us]: FAILED!
=> {"msg": "The conditional check 'etcd_image !=
l_default_osm_etcd_image' failed. The error was: An unhandled exception occurred while
templating '{{ osm_etcd_image }}'. Error was a <class
'ansible.errors.AnsibleError'>, original message: An unhandled exception
occurred while templating '{{ l_default_osm_etcd_image }}'. Error was a <class
'ansible.errors.AnsibleError'>, original message: An unhandled exception
occurred while templating '{{ etcd_image_dict[openshift_deployment_type] |
lib_utils_oo_oreg_image((oreg_url | default('None'))) }}'. Error was a
<class 'ansible.errors.AnsibleFilterError'>, original message: oreg_url
malformed: registry.redhat.io\n\nThe error appears to have been in
'/usr/share/ansible/openshift-ansible/roles/etcd/tasks/static.yml': line 5, column
3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe
offending line appears to be:\n\n\n- name:
Warn if osm_etcd_image is redefined\n ^ here\n"}
2019-05-28 20:07:12,688 p=22689 u=root | NO MORE HOSTS LEFT
**************************************************************************************************************************************************************************************************************************
2019-05-28 20:07:12,689 p=22689 u=root | PLAY RECAP
**********************************************************************************************************************************************************************************************************************************
2019-05-28 20:07:12,689 p=22689 u=root | localhost : ok=35 changed=1
unreachable=0 failed=0
2019-05-28 20:07:12,689 p=22689 u=root | master0.ent.co.ventura.ca.us : ok=204
changed=16 unreachable=0 failed=1
2019-05-28 20:07:12,690 p=22689 u=root | INSTALLER STATUS
****************************************************************************************************************************************************************************************************************************
2019-05-28 20:07:12,693 p=22689 u=root | Initialization : Complete
(0:00:10)
2019-05-28 20:07:12,693 p=22689 u=root | Health Check : Complete
(0:00:22)
2019-05-28 20:07:12,694 p=22689 u=root | Node Bootstrap Preparation : Complete
(0:01:59)
2019-05-28 20:07:12,694 p=22689 u=root | etcd Install : In Progress
(0:00:28)
2019-05-28 20:07:12,694 p=22689 u=root | This phase can be restarted by running:
playbooks/openshift-etcd/config.yml
2019-05-28 20:07:12,694 p=22689 u=root | Tuesday 28 May 2019 20:07:12 -0400
(0:00:00.044) 0:04:54.819 ***********
2019-05-28 20:07:12,694 p=22689 u=root |
===============================================================================
2019-05-28 20:07:12,697 p=22689 u=root | openshift_node : install needed rpm(s)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
38.86s
2019-05-28 20:07:12,697 p=22689 u=root | Ensure openshift-ansible installer package deps
are installed
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
36.95s
2019-05-28 20:07:12,697 p=22689 u=root | openshift_node : Install node, clients, and
conntrack packages
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
23.81s
2019-05-28 20:07:12,698 p=22689 u=root | Run health checks (install) - EL
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
22.26s
2019-05-28 20:07:12,698 p=22689 u=root | container_runtime : Install Docker
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
7.10s
2019-05-28 20:07:12,698 p=22689 u=root | openshift_repos : Disable all repositories
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6.78s
2019-05-28 20:07:12,698 p=22689 u=root | openshift_repos : Enable RHEL repositories
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6.09s
2019-05-28 20:07:12,698 p=22689 u=root | openshift_node : Create credentials for registry
auth
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
5.57s
2019-05-28 20:07:12,698 p=22689 u=root | rhel_subscribe : Install Red Hat Subscription
manager
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
5.22s
2019-05-28 20:07:12,698 p=22689 u=root | openshift_repos : Ensure libselinux-python is
installed
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
5.09s
2019-05-28 20:07:12,698 p=22689 u=root | nickhammond.logrotate : nickhammond.logrotate |
Install logrotate
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.99s
2019-05-28 20:07:12,699 p=22689 u=root | etcd : Install openssl
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.79s
2019-05-28 20:07:12,699 p=22689 u=root | openshift_node : Install dnsmasq
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.68s
2019-05-28 20:07:12,699 p=22689 u=root | etcd : Install openssl
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.68s
2019-05-28 20:07:12,699 p=22689 u=root | os_firewall : Install firewalld packages
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.63s
2019-05-28 20:07:12,699 p=22689 u=root | install NetworkManager
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.59s
2019-05-28 20:07:12,699 p=22689 u=root | etcd : Install etcd
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.57s
2019-05-28 20:07:12,699 p=22689 u=root | container_runtime : Create credentials for
oreg_url
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.54s
2019-05-28 20:07:12,700 p=22689 u=root | openshift_node : Install NFS storage plugin
dependencies
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
4.53s
2019-05-28 20:07:12,700 p=22689 u=root | openshift_node : Add firewalld allow rules
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3.39s
2019-05-28 20:07:12,700 p=22689 u=root | Failure summary:
1. Hosts: master0.ent.co.ventura.ca.us
Play: Configure etcd
Task: Warn if osm_etcd_image is redefined
Message: [0;31mThe conditional check 'etcd_image !=
l_default_osm_etcd_image' failed. The error was: An unhandled exception occurred while
templating '{{ osm_etcd_image }}'. Error was a <class
'ansible.errors.AnsibleError'>, original message: An unhandled exception
occurred while templating '{{ l_default_osm_etcd_image }}'. Error was a <class
'ansible.errors.AnsibleError'>, original message: An unhandled exception
occurred while templating '{{ etcd_image_dict[openshift_deployment_type] |
lib_utils_oo_oreg_image((oreg_url | default('None'))) }}'. Error was a
<class 'ansible.errors.AnsibleFilterError'>, original message: oreg_url
malformed: registry.redhat.io[0m
[0;31m[0m
[0;31mThe error appears to have been in
'/usr/share/ansible/openshift-ansible/roles/etcd/tasks/static.yml': line 5, column
3, but may[0m
[0;31mbe elsewhere in the file depending on the exact syntax problem.[0m
[0;31m[0m
[0;31mThe offending line appears to be:[0m
[0;31m[0m
[0;31m[0m
[0;31m- name: Warn if osm_etcd_image is redefined[0m
[0;31m ^ here[0m
[0;31m[0m
12:10 p.m.
Hi,
Is it possible that you uncommented the oreg_url?
I see this in the error message:
oreg_url malformed: registry.redhat.io
It should usually stay as :
# oreg_url: registry.redhat.io
--
Shirly Radco
BI Senior Software Engineer
Red Hat <https://www.redhat.com/>
<https://www.redhat.com/>
On Wed, May 29, 2019 at 7:41 PM <roy.morris(a)ventura.org> wrote:
12:18 p.m.
We currently use RHEV and I was under the impression I had to login to pull the
openshift/docker images. I could be wrong since I'm new to this.
1:18 p.m.
When using RHV thr only difference is that you need to provide the
subscription credentials and all are pulled automatically.
--
Shirly Radco
BI Senior Software Engineer
Red Hat <https://www.redhat.com/>
<https://www.redhat.com/>
On Wed, May 29, 2019 at 8:21 PM <roy.morris(a)ventura.org> wrote:
12:08 p.m.
I have removed the metrics-installer and master0 VM in addition to commenting out the #
oreg_url: registry.redhat.io line.
I'm now stuck at cockpit-docker not being installed. I feel like I'm so close and
I appreciate all the help I can get.
TASK [openshift_manageiq : Configure 3_2 role/user permissions] ****************
Wednesday 29 May 2019 19:37:23 -0400 (0:00:06.860) 0:27:45.295 *********
changed: [master0.ent.co.ventura.ca.us] => (item={u'resource_kind':
u'cluster-role', u'resource_name': u'system:image-auditor',
u'user': u'system:serviceaccount:management-infra:management-admin'})
TASK [cockpit : Add iptables allow rules] **************************************
Wednesday 29 May 2019 19:37:24 -0400 (0:00:00.937) 0:27:46.233 *********
skipping: [master0.ent.co.ventura.ca.us] => (item={u'port':
u'9090/tcp', u'service': u'cockpit-ws'})
TASK [cockpit : Remove iptables rules] *****************************************
Wednesday 29 May 2019 19:37:24 -0400 (0:00:00.137) 0:27:46.370 *********
TASK [cockpit : Add firewalld allow rules] *************************************
Wednesday 29 May 2019 19:37:24 -0400 (0:00:00.066) 0:27:46.437 *********
changed: [master0.ent.co.ventura.ca.us] => (item={u'port': u'9090/tcp',
u'service': u'cockpit-ws'})
TASK [cockpit : Remove firewalld allow rules] **********************************
Wednesday 29 May 2019 19:37:24 -0400 (0:00:00.785) 0:27:47.223 *********
TASK [cockpit : Install cockpit-ws] ********************************************
Wednesday 29 May 2019 19:37:25 -0400 (0:00:00.069) 0:27:47.292 *********
FAILED - RETRYING: Install cockpit-ws (3 retries left).
FAILED - RETRYING: Install cockpit-ws (2 retries left).
FAILED - RETRYING: Install cockpit-ws (1 retries left).
fatal: [master0.ent.co.ventura.ca.us]: FAILED! => {"attempts": 3,
"changed": false, "msg": "No package matching
'cockpit-docker' found available, installed or updated", "rc": 126,
"results": ["No package matching 'cockpit-docker' found available,
installed or updated"]}
PLAY RECAP *********************************************************************
localhost : ok=35 changed=1 unreachable=0 failed=0
master0.ent.co.ventura.ca.us : ok=517 changed=234 unreachable=0 failed=1
INSTALLER STATUS ***************************************************************
Initialization : Complete (0:00:09)
Health Check : Complete (0:01:06)
Node Bootstrap Preparation : Complete (0:10:30)
etcd Install : Complete (0:01:52)
Master Install : Complete (0:05:46)
Master Additional Install : In Progress (1:48:51)
This phase can be restarted by running:
playbooks/openshift-master/additional_config.yml
Wednesday 29 May 2019 21:25:49 -0400 (1:48:24.467) 2:16:11.760 *********
===============================================================================
cockpit : Install cockpit-ws ----------------------------------------- 6504.47s
openshift_node : install needed rpm(s) -------------------------------- 450.47s
container_runtime : Install Docker ------------------------------------ 234.94s
openshift_node : Install node, clients, and conntrack packages --------- 85.65s
Ensure openshift-ansible installer package deps are installed ---------- 82.19s
openshift_control_plane : Wait for all control plane pods to become ready -- 78.19s
Run health checks (install) - EL --------------------------------------- 65.70s
etcd : Install etcd ---------------------------------------------------- 65.53s
openshift_control_plane : Wait for control plane pods to appear -------- 47.04s
openshift_ca : Install the base package for admin tooling -------------- 41.98s
os_firewall : Install firewalld packages ------------------------------- 41.67s
openshift_node : Install dnsmasq --------------------------------------- 33.34s
openshift_repos : Discover enabled RHEL rhui repositories -------------- 15.23s
openshift_cli : Install clients ---------------------------------------- 13.07s
openshift_node_group : Wait for the sync daemonset to become ready and available --
11.08s
rhel_subscribe : Register host using user/password --------------------- 10.78s
os_firewall : need to pause here, otherwise the firewalld service starting can sometimes
cause ssh to fail -- 10.24s
openshift_repos : Disable all repositories ------------------------------ 8.90s
openshift_repos : Enable RHEL repositories ------------------------------ 7.69s
openshift_manageiq : Configure role/user permissions -------------------- 6.86s
Failure summary:
1. Hosts: master0.ent.co.ventura.ca.us
Play: Additional master configuration
Task: Install cockpit-ws
Message: No package matching 'cockpit-docker' found available, installed or
updated
2:50 p.m.
That did the trick! However, now I'm stuck at Install cockpit-ws. It keeps failing. I
noticed yum update fails to find mirrors as well.
1. Hosts: master0.ent.co.ventura.ca.us
Play: Additional master configuration
Task: Install cockpit-ws
Message: Failure talking to yum: failure: repodata/repomd.xml from
rhel-7-server-rpms: [Errno 256] No more mirrors to try.
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repod...:
[Errno 14] curl#6 - "Could not resolve host: cdn.redhat.com; Unknown error"
6:47 p.m.
I ran this command on the master0 VM and this is a fresh install I performed yesterday
after commenting out the last variable issue I experienced. To me this is an issue
somewhere in the playbook since I haven't customized anything except setting IP
settings. I really want to get this ticket moving and I need active help please. This
looks to be a bug.
#systemctl status selinux* -l
selinux-policy-migrate-local-changes(a)targeted.service - Migrate local SELinux policy
changes from the old store structure to the new structure
Loaded: loaded
(/usr/lib/systemd/system/basic.target.wants/../selinux-policy-migrate-local-changes@.service;
static; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2019-05-29 18:59:22 EDT; 24h ago
Main PID: 3376 (code=exited, status=208/STDIN)
May 29 18:59:22 localhost systemd[1]: Starting Migrate local SELinux policy changes from
the old store structure to the new structure...
May 29 18:59:22 localhost systemd[3376]: Failed at step STDIN spawning
/usr/libexec/selinux/selinux-policy-migrate-local-changes.sh: Inappropriate ioctl for
device
May 29 18:59:22 localhost systemd[1]:
selinux-policy-migrate-local-changes(a)targeted.service: main process exited, code=exited,
status=208/STDIN
May 29 18:59:22 localhost systemd[1]: Failed to start Migrate local SELinux policy changes
from the old store structure to the new structure.
May 29 18:59:22 localhost systemd[1]: Unit
selinux-policy-migrate-local-changes(a)targeted.service entered failed state.
May 29 18:59:22 localhost systemd[1]:
selinux-policy-migrate-local-changes(a)targeted.service failed.
6:47 p.m.
I ran this command on the master0 VM and this is a fresh install I performed yesterday
after commenting out the last variable issue I experienced. To me this is an issue
somewhere in the playbook since I haven't customized anything except setting IP
settings. This looks to be a bug.
#systemctl status selinux* -l
selinux-policy-migrate-local-changes(a)targeted.service - Migrate local SELinux policy
changes from the old store structure to the new structure
Loaded: loaded
(/usr/lib/systemd/system/basic.target.wants/../selinux-policy-migrate-local-changes@.service;
static; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2019-05-29 18:59:22 EDT; 24h ago
Main PID: 3376 (code=exited, status=208/STDIN)
May 29 18:59:22 localhost systemd[1]: Starting Migrate local SELinux policy changes from
the old store structure to the new structure...
May 29 18:59:22 localhost systemd[3376]: Failed at step STDIN spawning
/usr/libexec/selinux/selinux-policy-migrate-local-changes.sh: Inappropriate ioctl for
device
May 29 18:59:22 localhost systemd[1]:
selinux-policy-migrate-local-changes(a)targeted.service: main process exited, code=exited,
status=208/STDIN
May 29 18:59:22 localhost systemd[1]: Failed to start Migrate local SELinux policy changes
from the old store structure to the new structure.
May 29 18:59:22 localhost systemd[1]: Unit
selinux-policy-migrate-local-changes(a)targeted.service entered failed state.
May 29 18:59:22 localhost systemd[1]:
selinux-policy-migrate-local-changes(a)targeted.service failed.
8:54 a.m.
Hi,
I passed the email to additional engineers for consultation.
Which IP did you update and in what stage?
Did you set the variable that sets the /etc/hosts records? If yes, Did you
update the records on the engine and vms?
Please attach your ansible log and config files so I can review them.
You can also open a bug and attach the to it, so we can track the issue
better.
Best,
Shirly
On Fri, May 31, 2019, 02:53 <roy.morris(a)ventura.org> wrote:
11:42 a.m.
Shirly,
I updated the IP of the metrics-store-installer VM once it was created. The script
continued to run after I updated the IP from the manager VM and completed successfully.
I also updated the master0 VM when the script got to the point of trying to contact it and
once I assigned the IP, it continued successfully.
---------
Okay, so here is the issue so far. I've narrowed it down to DNS resolution wanting to
go through tun0 adapter rather than eth0 which happens at the end of the playbook. It all
works fine and routes correctly when running the playbook up until something causes routes
to change and DNS tries to go through tun0.
I figured this out by opening two terminal windows and running a ping command inside
master0 VM towards redhat.com and in the other terminal window on master0 VM I ran #ip
monitor
The result is as mentioned. DNS entries are attempting to route through tun0 rather than
eth0
3:47 a.m.
Hi,
Please see here for the fix that caused the nics issue
https://gerrit.ovirt.org/#/c/100865/
The /etc/ovirt-guest-agent.conf ignored_nics should be update to
ignored_nics = docker0 tun0
Best,
--
Shirly Radco
BI Senior Software Engineer
Red Hat <https://www.redhat.com/>
<https://www.redhat.com/>
On Mon, Jun 3, 2019 at 7:44 PM <roy.morris(a)ventura.org> wrote:
2121
days inactive
2140
days old
11 comments
2 participants
participants (2)
-
roy.morris＠ventura.org -
Shirly Radco