HE fails to deploy at waiting for host to be up in the local HE VM. The setup logs does not indicate why it failed - atleast I couldn't find anything

---------- Forwarded message ---------- From: <jenkins(a)jenkins.phx.ovirt.org&gt; Date: Mon, Apr 2, 2018 at 7:50 PM Subject: [oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 276 - Still Failing! To: infra(a)ovirt.org, sabose(a)redhat.com Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-sui te-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-sui te-master/276/ Build Number: 276 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #265 [Gal Ben Haim] Check if the prefix exists before printing its size [Sandro Bonazzola] ovirt-engine: add jobs for 4.1.10 async Changes for Build #266 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #267 [Gal Ben Haim] Check if the prefix exists before printing its size [Daniel Belenky] ppc repos: Use qemu EV release instead of test [Daniel Belenky] global_setup: Add generic package remove function [Daniel Belenky] Fix package verification in verify_packages Changes for Build #268 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #269 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #270 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #271 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #272 [Gal Ben Haim] Check if the prefix exists before printing its size Changes for Build #273 [Eitan Raviv] network: macpool: test disallowing dups while dups exist [Daniel Belenky] docker cleanup:Fix edge case for unamed containers [Daniel Belenky] nested_config: Count nesting level of options [Daniel Belenky] Introduce conditional execution in STDCI DSL [Daniel Belenky] Add OST STDCI V2 jobs Changes for Build #274 [Gal Ben Haim] he-iscsi-master: Temporarily exclude in check-patch Changes for Build #275 [Gal Ben Haim] he-iscsi-master: Temporarily exclude in check-patch Changes for Build #276 [Barak Korren] Force STDCI V2 job to use physical host [Daniel Belenky] Build container on changes to docker_cleanup ----------------- Failed Tests: ----------------- No tests ran. _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On Tue, Apr 3, 2018 at 10:14 AM, Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Apr 2, 2018 at 4:44 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > >> HE fails to deploy at waiting for host to be up in the local HE VM. >> The setup logs does not indicate why it failed - atleast I couldn't find >> anything >> > > I see: > > "status": "install_failed" > > So I think that something went wrong with host-deploy on that host but we > definitively need host-deploy logs for that and they are just on the engine > VM. > According to the timestamps it could be related to: Apr 2 09:58:13 lago-hc-basic-suite-master-host-0 systemd: Starting Open vSwitch Database Unit... Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 ovs-ctl: runuser: System error Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 ovs-ctl: /etc/openvswitch/conf.db does not exist ... (warning). Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 ovs-ctl: Creating empty database /etc/openvswitch/conf.db runuser: System error Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 ovs-ctl: [FAILED] Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: ovsdb-server.service: control process exited, code=exited status=1 Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Failed to start Open vSwitch Database Unit. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Unit ovsdb-server.service entered failed state. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: ovsdb-server.service failed. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Invalid request descriptor Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Assertion failed for Open vSwitch Delete Transient Ports. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: ovsdb-server.service holdoff time over, scheduling restart. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: start request repeated too quickly for ovsdb-server.service Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Failed to start Open vSwitch Database Unit. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: Unit ovsdb-server.service entered failed state. Apr 2 09:58:14 lago-hc-basic-suite-master-host-0 systemd: ovsdb-server.service failed.

> > >> >> ---------- Forwarded message ---------- >> From: <jenkins(a)jenkins.phx.ovirt.org&gt; >> Date: Mon, Apr 2, 2018 at 7:50 PM >> Subject: [oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - >> Build # 276 - Still Failing! >> To: infra(a)ovirt.org, sabose(a)redhat.com >> >> >> Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-sui >> te-master/ >> Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-sui >> te-master/276/ >> Build Number: 276 >> Build Status: Still Failing >> Triggered By: Started by timer >> >> ------------------------------------- >> Changes Since Last Success: >> ------------------------------------- >> Changes for Build #265 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> [Sandro Bonazzola] ovirt-engine: add jobs for 4.1.10 async >> >> >> Changes for Build #266 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> >> Changes for Build #267 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> [Daniel Belenky] ppc repos: Use qemu EV release instead of test >> >> [Daniel Belenky] global_setup: Add generic package remove function >> >> [Daniel Belenky] Fix package verification in verify_packages >> >> >> Changes for Build #268 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> >> Changes for Build #269 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> >> Changes for Build #270 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> >> Changes for Build #271 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> >> Changes for Build #272 >> [Gal Ben Haim] Check if the prefix exists before printing its size >> >> >> Changes for Build #273 >> [Eitan Raviv] network: macpool: test disallowing dups while dups exist >> >> [Daniel Belenky] docker cleanup:Fix edge case for unamed containers >> >> [Daniel Belenky] nested_config: Count nesting level of options >> >> [Daniel Belenky] Introduce conditional execution in STDCI DSL >> >> [Daniel Belenky] Add OST STDCI V2 jobs >> >> >> Changes for Build #274 >> [Gal Ben Haim] he-iscsi-master: Temporarily exclude in check-patch >> >> >> Changes for Build #275 >> [Gal Ben Haim] he-iscsi-master: Temporarily exclude in check-patch >> >> >> Changes for Build #276 >> [Barak Korren] Force STDCI V2 job to use physical host >> >> [Daniel Belenky] Build container on changes to docker_cleanup >> >> >> >> >> ----------------- >> Failed Tests: >> ----------------- >> No tests ran. >> >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > >

2018-04-05 20:46:52,773-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:100 TASK [Get local VM IP] 2018-04-05 20:55:28,217-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2018-04-05 20:55:28.046320', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'start': u'2018-04-05 20:55:28.000470', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.045850', u'stdout_lines': []} 2018-04-05 20:55:28,318-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.045850", "end": "2018-04-05 20:55:28.046320", "rc": 0, "start": "2018-04-05 20:55:28.000470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Both the 4.2 and master suites are failing on getting local VM IP. Any idea what changed or if I have to change the test? thanks!

On Fri, Apr 6, 2018 at 1:10 PM, Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Fri, Apr 6, 2018 at 9:28 AM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > >> 2018-04-05 20:46:52,773-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:100 TASK [Get local VM IP] >> 2018-04-05 20:55:28,217-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2018-04-05 20:55:28.046320', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'start': u'2018-04-05 20:55:28.000470', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.045850', u'stdout_lines': []} >> 2018-04-05 20:55:28,318-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.045850", "end": "2018-04-05 20:55:28.046320", "rc": 0, "start": "2018-04-05 20:55:28.000470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} >> >> Both the 4.2 and master suites are failing on getting local VM IP. >> Any idea what changed or if I have to change the test? >> >> thanks! >> > > Hi Sahina, > 4.2 and master suite non HC are correctly running this morning. > http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovi > rt-system-tests_he-basic-ansible-suite-master/146/ > http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovi > rt-system-tests_he-basic-ansible-suite-4.2/76/ > > I'll try to check the difference with HC suites. > > Are you using more than one subnet in the HC suites? > No, I'm not. And we havent's changed anything related to network in the test suite.