[ovirt-devel] Re: [ OST Failure Report ] [ oVirt master ] [ 2018-05-25 ] [AddHost]

+Martin He is working on it. Thanks, Piotr On Tue, May 29, 2018 at 2:22 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: > Hi Piotr, > > Any update on this? > > Thanks. > Dafna > > > On Mon, May 28, 2018 at 10:59 AM, Piotr Kliczewski < > piotr.kliczewski(a)gmail.com&gt; wrote: > >> On Mon, May 28, 2018 at 11:41 AM, Barak Korren <bkorren(a)redhat.com&gt; >> wrote: >> > >> > >> > On 28 May 2018 at 12:38, Piotr Kliczewski <piotr.kliczewski(a)gmail.com&gt; >> > wrote: >> >> >> >> On Mon, May 28, 2018 at 10:57 AM, Barak Korren <bkorren(a)redhat.com&gt; >> wrote: >> >> > Note: we're now seeing a very similar issue in the 4.2 branch as >> well >> >> > that >> >> > seems to have been introduced by the following patch: >> >> >> >> Can you point to specific job so we could take a look at the logs? >> > >> > >> > Whoops, sorry, here: >> > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2034/ >> > >> >> Looks like the same issue: >> >> 2018-05-28 03:41:03,606-04 ERROR >> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >> (EE-ManagedThreadFactory-engine-Thread-1) [1244c90f] SSH error running >> command root@lago-upgrade-from-prevrelease-suite-4-2-host-0:'umask >> 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t >> ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null >> 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar >> --warning=no-timestamp -C "${MYTMP}" -x && >> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >> DIALOG/customization=bool:True': TimeLimitExceededException: SSH >> session timeout host >> 'root@lago-upgrade-from-prevrelease-suite-4-2-host-0' >> 2018-05-28 03:41:03,606-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) >> [1244c90f] Error during deploy dialog >> 2018-05-28 03:41:03,611-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [1244c90f] Timeout during >> host lago-upgrade-from-prevrelease-suite-4-2-host-0 install: SSH >> session timeout host >> 'root@lago-upgrade-from-prevrelease-suite-4-2-host-0' >> >> >> >> >> >> >> > >> >> > https://gerrit.ovirt.org/c/91638/2 - core: Enable only strong >> ciphers >> >> > for >> >> > 4.2 hosts >> >> > >> >> > On 28 May 2018 at 10:26, Barak Korren <bkorren(a)redhat.com&gt; wrote: >> >> >> >> >> >> >> >> >> >> >> >> On 28 May 2018 at 10:19, Martin Perina <mperina(a)redhat.com&gt; wrote: >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Mon, May 28, 2018 at 9:00 AM, Piotr Kliczewski >> >> >>> <pkliczew(a)redhat.com&gt; >> >> >>> wrote: >> >> >>>> >> >> >>>> Simone, >> >> >>>> >> >> >>>> What do you think about this failure? >> >> >>>> >> >> >>>> Thanks, >> >> >>>> Piotr >> >> >>>> >> >> >>>> On Mon, May 28, 2018 at 7:12 AM, Barak Korren < >> bkorren(a)redhat.com&gt; >> >> >>>> wrote: >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> On 27 May 2018 at 14:59, Piotr Kliczewski <pkliczew(a)redhat.com&gt; >> >> >>>>> wrote: >> >> >>>>>> >> >> >>>>>> Martin, >> >> >>>>>> >> >> >>>>>> I only can see: >> >> >>>>>> >> >> >>>>>> 2018-05-25 13:57:44,255-04 ERROR >> >> >>>>>> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >> >> >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [55a7b15b] SSH error >> >> >>>>>> running >> >> >>>>>> command root@lago-upgrade-from-release >> -suite-master-host-0:'umask >> >> >>>>>> 0077; >> >> >>>>>> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t >> ovirt-XXXXXXXXXX)"; >> >> >>>>>> trap >> >> >>>>>> "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr >> \"${MYTMP}\" >> >> >>>>>> > >> >> >>>>>> /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x >> && >> >> >>>>>> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >> >> >>>>>> DIALOG/customization=bool:True': TimeLimitExceededException: >> SSH >> >> >>>>>> session >> >> >>>>>> timeout host 'root@lago-upgrade-from-releas >> e-suite-master-host-0' >> >> >>>>>> 2018-05-25 13:57:44,259-04 ERROR >> >> >>>>>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> >> >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [55a7b15b] Timeout >> during >> >> >>>>>> host >> >> >>>>>> lago-upgrade-from-release-suite-master-host-0 install: SSH >> session >> >> >>>>>> timeout >> >> >>>>>> host 'root@lago-upgrade-from-release-suite-master-host-0' >> >> >>>>>> >> >> >>>>>> There are no additional logs. SSH to host timeout. Are we sure >> that >> >> >>>>>> it >> >> >>>>>> is an issue caused by Ravi's change? >> >> >>>>> >> >> >>>>> >> >> >>>>> We have some quite strong circumstantial evidence: >> >> >>>>> - Issue had affected all engine patches since that patch in a >> >> >>>>> similar >> >> >>>>> fashion. >> >> >>>>> - Prior engine patch [1] passed successfully [2] >> >> >>>>> - Other subsequent OST runs without engine patches passed >> >> >>>>> successfully >> >> >>>>> as well [3]. >> >> >>>>> >> >> >>>>> [1]: https://gerrit.ovirt.org/c/91595/2 >> >> >>>>> [2]: >> >> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >> r/7777/ >> >> >>>>> [3]: >> >> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >> r/7778/ >> >> >>>>> >> >> >>>>> >> >> >>>>> Please note - the issue is affecting a test that is run by an >> >> >>>>> upgrade >> >> >>>>> suit on the post-upgrade system. It has no affect on the basic >> suit. >> >> >>>>> So it >> >> >>>>> probably has to do with some behaviour that is specific to >> upgraded >> >> >>>>> systems. >> >> >>> >> >> >>> >> >> >>> I will try to reproduce later today in dev env, but I agree with >> >> >>> Piotr's >> >> >>> investigation, engine was not able to connect to the host using >> SSH >> >> >>> and >> >> >>> that's why no host-deploy logs were fetched. >> >> >> >> >> >> >> >> >> Lago fetches the logs from the host too (And it can take then from >> the >> >> >> VM >> >> >> image directly if the host is not responsive over SSH), can we get >> at >> >> >> the >> >> >> host-deploy logs that way? >> >> >> >> >> >> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> Thanks, >> >> >>>>>> Piotr >> >> >>>>>> >> >> >>>>>> On Sun, May 27, 2018 at 11:21 AM, Martin Perina >> >> >>>>>> <mperina(a)redhat.com&gt; >> >> >>>>>> wrote: >> >> >>>>>>> >> >> >>>>>>> Adding also Piotr to the thread >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> On Sun, 27 May 2018, 08:46 Barak Korren, <bkorren(a)redhat.com&gt; >> >> >>>>>>> wrote: >> >> >>>>>>>> >> >> >>>>>>>> Test failed: [ AddHost (in upgrade-from-release-suite) ] >> >> >>>>>>>> >> >> >>>>>>>> Link to suspected patches: >> >> >>>>>>>> https://gerrit.ovirt.org/#/c/91445/5 - Disable TLS versions >> < 1.2 >> >> >>>>>>>> for hosts with cluster level>=4.1 >> >> >>>>>>>> >> >> >>>>>>>> Link to Job: >> >> >>>>>>>> >> >> >>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >> r/7776/ >> >> >>>>>>>> >> >> >>>>>>>> Link to all logs: >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >> r/7776/artifact/exported-artifacts/upgrade-from-release-suit >> -master-el7/test_logs/upgrade-from-release-suite-master/ >> post-002_bootstrap.py/ >> >> >>>>>>>> >> >> >>>>>>>> Error snippet from log: >> >> >>>>>>>> >> >> >>>>>>>> From nosetst log: >> >> >>>>>>>> <error> >> >> >>>>>>>> >> >> >>>>>>>> AssertionError: False != True after 1200 seconds >> >> >>>>>>>> >> >> >>>>>>>> </error> >> >> >>>>>>>> >> >> >>>>>>>> Not finding a host deploy log in /var/log/ovirt-engine for >> some >> >> >>>>>>>> reason. >> >> >>>>>>>> This seems to have cause consistent failure in all other >> engine >> >> >>>>>>>> patches that followed it. >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> -- >> >> >>>>>>>> Barak Korren >> >> >>>>>>>> RHV DevOps team , RHCE, RHCi >> >> >>>>>>>> Red Hat EMEA >> >> >>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> >> >>>>>> >> >> >>>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> -- >> >> >>>>> Barak Korren >> >> >>>>> RHV DevOps team , RHCE, RHCi >> >> >>>>> Red Hat EMEA >> >> >>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> >> >>>> >> >> >>>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Martin Perina >> >> >>> Associate Manager, Software Engineering >> >> >>> Red Hat Czech s.r.o. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Barak Korren >> >> >> RHV DevOps team , RHCE, RHCi >> >> >> Red Hat EMEA >> >> >> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > Barak Korren >> >> > RHV DevOps team , RHCE, RHCi >> >> > Red Hat EMEA >> >> > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> >> > >> >> > _______________________________________________ >> >> > Devel mailing list -- devel(a)ovirt.org >> >> > To unsubscribe send an email to devel-leave(a)ovirt.org >> >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> >> > oVirt Code of Conduct: >> >> > https://www.ovirt.org/community/about/community-guidelines/ >> >> > List Archives: >> >> > >> >> > https://lists.ovirt.org/archives/list/devel@ovirt.org/messag >> e/QIZ5L4FKII7X5FHQ4OXBBR2SLUIK5C74/ >> >> > >> > >> > >> > >> > >> > -- >> > Barak Korren >> > RHV DevOps team , RHCE, RHCi >> > Red Hat EMEA >> > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/communit >> y/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archiv >> es/list/devel(a)ovirt.org/message/RDK42TYJKMX3M2DNUFKZO7CGNNOYWMJI/ >> > >