[ovirt-devel] Update: HC suites failing for 3 weeks ( was: [OST][HC] HE fails to deploy )

Mon Apr 30 13:13:27 UTC 2018

On Mon, Apr 30, 2018 at 3:59 PM, Sahina Bose <sabose at redhat.com> wrote:

>
>
> On Mon, Apr 30, 2018 at 5:45 PM, Eyal Edri <eedri at redhat.com> wrote:
>
>>
>>
>> On Wed, Apr 25, 2018 at 1:53 PM, Sahina Bose <sabose at redhat.com> wrote:
>>
>>>
>>>
>>> On Wed, Apr 25, 2018 at 3:54 PM, Sahina Bose <sabose at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Apr 23, 2018 at 6:28 PM, Sahina Bose <sabose at redhat.com> wrote:
>>>>
>>>>>
>>>>> On Mon, Apr 23, 2018 at 5:41 PM, Eyal Edri <eedri at redhat.com> wrote:
>>>>>
>>>>>> Sahina,
>>>>>> Any update on this?
>>>>>>
>>>>>
>>>>> Sorry, haven't been able to spend any time on this. The last I checked
>>>>> the  HE install was failing at task - Get Local VM IP.
>>>>> and there were no logs from HE VM to debug.
>>>>>
>>>>> Will spend sometime on this tomorrow
>>>>>
>>>>
>>>> https://gerrit.ovirt.org/#/c/89953/ - fixes the issue, atleast when I
>>>> tried this on my local setup.
>>>>
>>>
>>>
>>> The CI however still fails in the HE install with :
>>>
>>> TASK [Get local VM IP]", "[ ERROR ] fatal: [localhost]: FAILED! => {\"attempts\": 50, \"changed\": true, \"cmd\": \"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'\", \"delta\": \"0:00:00.043961\", \"end\": \"2018-04-25 05:51:34.226374\", \"rc\": 0, \"start\": \"2018-04-25 05:51:34.182413\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"\", \"stdout_lines\": []}"
>>>
>>>
>>>
>>> FWIW, my local setup , ost repo was at I3fc2976ab2400e5908760aadc3258
>>> 329c0ffdf4d
>>>
>>
>>
>> Any update? suites are still failing.
>>
>
> The suites work locally, on the CI systems they fail with above error.
> Unfortunately, no clue as to why this is so.
> With the current run, are we able to get logs from the engine VM?
>

The logs should be in the job build-artifacts [1]

[1]
http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/122/consoleFull

>
>
>>
>>>
>>>
>>>>
>>>>
>>>>>
>>>>>> On Wed, Apr 18, 2018 at 3:40 PM, Sandro Bonazzola <
>>>>>> sbonazzo at redhat.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2018-04-18 9:37 GMT+02:00 Eyal Edri <eedri at redhat.com>:
>>>>>>>
>>>>>>>> FYI,
>>>>>>>>
>>>>>>>> I've disabled the 4.2 and master HC suites nightly run on CI as
>>>>>>>> they are constantly failing for almost 3 weeks and spamming the mailing
>>>>>>>> lists.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> HC uses gdeploy 2.0.6 which was released in December and was based
>>>>>>> on ansible 2.4.
>>>>>>> ansible-2.5 landed 3 weeks ago in EPEL, my guess is that gdeploy is
>>>>>>> not supporting ansible-2.5 properly.
>>>>>>> I had no time to validate my guess with proof, so please Sahina
>>>>>>> cross check this.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I think this should get higher priority for a fix if we want it to
>>>>>>>> provide any value,
>>>>>>>> Work can continue using the manual jobs or via check-patch.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Apr 16, 2018 at 10:56 AM, Gal Ben Haim <gbenhaim at redhat.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Any update on https://gerrit.ovirt.org/#/c/88887/ ?
>>>>>>>>> The HC suites still failing and it's hard to understand why
>>>>>>>>> without the logs from the engine VM.
>>>>>>>>>
>>>>>>>>> On Sat, Apr 7, 2018 at 7:19 AM, Sahina Bose <sabose at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 6, 2018 at 1:10 PM, Simone Tiraboschi <
>>>>>>>>>> stirabos at redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 6, 2018 at 9:28 AM, Sahina Bose <sabose at redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 2018-04-05 20:46:52,773-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:100 TASK [Get local VM IP]
>>>>>>>>>>>> 2018-04-05 20:55:28,217-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2018-04-05 20:55:28.046320', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'start': u'2018-04-05 20:55:28.000470', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.045850', u'stdout_lines': []}
>>>>>>>>>>>> 2018-04-05 20:55:28,318-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:24:d3:63 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.045850", "end": "2018-04-05 20:55:28.046320", "rc": 0, "start": "2018-04-05 20:55:28.000470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
>>>>>>>>>>>>
>>>>>>>>>>>> Both the 4.2 and master suites are failing on getting local VM
>>>>>>>>>>>> IP.
>>>>>>>>>>>> Any idea what changed or if I have to change the test?
>>>>>>>>>>>>
>>>>>>>>>>>> thanks!
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Sahina,
>>>>>>>>>>> 4.2 and master suite non HC are correctly running this morning.
>>>>>>>>>>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovi
>>>>>>>>>>> rt-system-tests_he-basic-ansible-suite-master/146/
>>>>>>>>>>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovi
>>>>>>>>>>> rt-system-tests_he-basic-ansible-suite-4.2/76/
>>>>>>>>>>>
>>>>>>>>>>> I'll try to check the difference with HC suites.
>>>>>>>>>>>
>>>>>>>>>>> Are you using more than one subnet in the HC suites?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> No, I'm not. And we havent's changed anything related to network
>>>>>>>>>> in the test suite.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *GAL bEN HAIM*
>>>>>>>>> RHV DEVOPS
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Eyal edri
>>>>>>>>
>>>>>>>>
>>>>>>>> MANAGER
>>>>>>>>
>>>>>>>> RHV DevOps
>>>>>>>>
>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>
>>>>>>>>
>>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>>>> <https://redhat.com/trusted>
>>>>>>>> phone: +972-9-7692018
>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Devel mailing list
>>>>>>>> Devel at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> SANDRO BONAZZOLA
>>>>>>>
>>>>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>>>>>>>
>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>
>>>>>>> sbonazzo at redhat.com
>>>>>>> <https://red.ht/sig>
>>>>>>> <https://redhat.com/summit>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Eyal edri
>>>>>>
>>>>>>
>>>>>> MANAGER
>>>>>>
>>>>>> RHV DevOps
>>>>>>
>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>
>>>>>>
>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>> <https://redhat.com/trusted>
>>>>>> phone: +972-9-7692018
>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Eyal edri
>>
>>
>> MANAGER
>>
>> RHV DevOps
>>
>> EMEA VIRTUALIZATION R&D
>>
>>
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>
>

-- 

Eyal edri

MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180430/194c76a8/attachment-0001.html>