On 3 Mar 2020, at 09:14, Barak Korren <bkorren(a)redhat.com&gt; wrote: It matters - because it would block sending the relevant builds to the CQ.

One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.

A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?

Why is this not failing in check-patch as well BTW?

On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: is this a better list? or no one cares? > On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: > > > >> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers(a)redhat.com <mailto:sdickers@redhat.com>> wrote: >> >> Hi, >> >> On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log: > > it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. > >> >> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har >> [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har >> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 >> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... >> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... >> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2....;: unexpected end of file". >> >> Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine. >> >> Help please! >> >> [1] - https://gerrit.ovirt.org/#/c/107309/ <https://gerrit.ovirt.org/#/c/107309/> >> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >> >> -- >> Scott Dickerson >> Senior Software Engineer >> RHV-M Engineering - UX Team >> Red Hat, Inc > _______________________________________________ Infra mailing list -- infra(a)ovirt.org <mailto:infra@ovirt.org> To unsubscribe send an email to infra-leave(a)ovirt.org <mailto:infra-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... <https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com/> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted>

On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek(a)redhat.com&gt; wrote: > > > On 3 Mar 2020, at 09:14, Barak Korren <bkorren(a)redhat.com&gt; wrote: > > It matters - because it would block sending the relevant builds to the CQ. > > > yes > > > One thing that could make YARN fail in CI but not in local mock is the > fact the we have HTTP_PROXY defined in the CI environment and pointing to a > Squid server. > > > A typical issue we see people having is when connecting to `localhost` > and ending up being blocked by the proxy. Please make sure the NO_PROXY env > var is set appropriately if that is the case? > > > good point, we can try that I guess. Weird thing is it succeeds some time. > Scott, can you try that? > @Sandro Bonazzola <sbonazzo(a)redhat.com&gt; tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358

> > Why is this not failing in check-patch as well BTW? > > > I think it doesn’t do any downloads, just checks the presence of > package.json. > > Thanks, > michal > > > On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < > michal.skrivanek(a)redhat.com&gt; wrote: > >> is this a better list? or no one cares? >> >> On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; >> wrote: >> >> >> >> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers(a)redhat.com&gt; wrote: >> >> Hi, >> >> On the merge phase on patch [1], both the el7 and fc30 build have been >> failing. Examples are [2] and [3]. I'm guessing there are environmental >> issues that I can't fix from the project's perspective. Typical example of >> the error from the log: >> >> >> it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and >> fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. >> >> >> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har >> [2020-03-01T15:55:19.475Z] + >> /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js >> install --pure-lockfile --har >> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 >> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... >> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... >> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " >> https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: >> unexpected end of file". >> >> Running the build in mock_runner locally targeted for el7, el8, fc29 and >> fc30 work just fine. >> >> Help please! >> >> [1] - https://gerrit.ovirt.org/#/c/107309/ >> [2] - >> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >> [3] - >> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >> >> >> -- >> Scott Dickerson >> Senior Software Engineer >> RHV-M Engineering - UX Team >> Red Hat, Inc >> >> >> >> _______________________________________________ >> Infra mailing list -- infra(a)ovirt.org >> To unsubscribe send an email to infra-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... >> > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > > >

On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson <sdickers(a)redhat.com <mailto:sdickers@redhat.com>> ha scritto: On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch(a)redhat.com <mailto:sgratch@redhat.com>> wrote: On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: > On 3 Mar 2020, at 09:14, Barak Korren <bkorren(a)redhat.com <mailto:bkorren@redhat.com>> wrote: > > It matters - because it would block sending the relevant builds to the CQ. yes > > One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server. > > A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case? good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that? @Sandro Bonazzola <mailto:sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358 <https://gerrit.ovirt.org/107358> I was just watching the jenkins build on his patch and it failed with exactly the same error as before. FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309 <https://gerrit.ovirt.org/107309>) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-) Yes, still failing on error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2....;: unexpected end of file". so even using a proxy is not helping at all. It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(

> > Why is this not failing in check-patch as well BTW? I think it doesn’t do any downloads, just checks the presence of package.json. Thanks, michal > > On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: > is this a better list? or no one cares? > >> On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek(a)redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: >> >> >> >>> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers(a)redhat.com <mailto:sdickers@redhat.com>> wrote: >>> >>> Hi, >>> >>> On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log: >> >> it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. >> >>> >>> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har >>> [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har >>> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 >>> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... >>> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... >>> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2....;: unexpected end of file". >>> >>> Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine. >>> >>> Help please! >>> >>> [1] - https://gerrit.ovirt.org/#/c/107309/ <https://gerrit.ovirt.org/#/c/107309/> >>> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >>> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >>> >>> -- >>> Scott Dickerson >>> Senior Software Engineer >>> RHV-M Engineering - UX Team >>> Red Hat, Inc >> > > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org <mailto:infra@ovirt.org> > To unsubscribe send an email to infra-leave(a)ovirt.org <mailto:infra-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... <https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com <http://redhat.com/> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted> -- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <mailto:sbonazzo@redhat.com> <https://www.redhat.com/>Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>

Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked. I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens. Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz " Regards, Scott [1] - https://gerrit.ovirt.org/107367 [2] - https://gerrit.ovirt.org/107381

On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek < michal.skrivanek(a)redhat.com&gt; wrote: > > > On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > > Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson < > sdickers(a)redhat.com&gt; ha scritto: > >> >> >> On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch(a)redhat.com&gt; wrote: >> >>> >>> >>> On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < >>> michal.skrivanek(a)redhat.com&gt; wrote: >>> >>>> >>>> >>>> On 3 Mar 2020, at 09:14, Barak Korren <bkorren(a)redhat.com&gt; wrote: >>>> >>>> It matters - because it would block sending the relevant builds to the >>>> CQ. >>>> >>>> >>>> yes >>>> >>>> >>>> One thing that could make YARN fail in CI but not in local mock is the >>>> fact the we have HTTP_PROXY defined in the CI environment and pointing to a >>>> Squid server. >>>> >>>> >>>> A typical issue we see people having is when connecting to `localhost` >>>> and ending up being blocked by the proxy. Please make sure the NO_PROXY env >>>> var is set appropriately if that is the case? >>>> >>>> >>>> good point, we can try that I guess. Weird thing is it succeeds some >>>> time. >>>> Scott, can you try that? >>>> >>> >>> >>> >>> @Sandro Bonazzola <sbonazzo(a)redhat.com&gt; tries that by using a specific >>> local proxy solution: https://gerrit.ovirt.org/107358 >>> >>> >>> >> I was just watching the jenkins build on his patch and it failed with >> exactly the same error as before. >> >> FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) >> only needs to do a total of 8 npm/yarnpkg repo downloads per >> build-artifacts run. So it isn't exactly asking for a huge effort. ;-) >> > > Yes, still failing on > > error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". > > so even using a proxy is not helping at all. > > It appears to affect everyone using yarn and common solution I found is > just retry on failure till it works :-( > > > how to use NO_PROXY? We don’t really need nor want any proxy > > > > > > > >> >> >>> >>>> >>>> Why is this not failing in check-patch as well BTW? >>>> >>>> >>>> I think it doesn’t do any downloads, just checks the presence of >>>> package.json. >>>> >>>> Thanks, >>>> michal >>>> >>>> >>>> On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < >>>> michal.skrivanek(a)redhat.com&gt; wrote: >>>> >>>>> is this a better list? or no one cares? >>>>> >>>>> On 2 Mar 2020, at 09:40, Michal Skrivanek < >>>>> michal.skrivanek(a)redhat.com&gt; wrote: >>>>> >>>>> >>>>> >>>>> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers(a)redhat.com&gt; wrote: >>>>> >>>>> Hi, >>>>> >>>>> On the merge phase on patch [1], both the el7 and fc30 build have >>>>> been failing. Examples are [2] and [3]. I'm guessing there are >>>>> environmental issues that I can't fix from the project's perspective. >>>>> Typical example of the error from the log: >>>>> >>>>> >>>>> it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and >>>>> fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. >>>>> >>>>> >>>>> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har >>>>> [2020-03-01T15:55:19.475Z] + >>>>> /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js >>>>> install --pure-lockfile --har >>>>> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 >>>>> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... >>>>> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... >>>>> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " >>>>> https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: >>>>> unexpected end of file". >>>>> >>>>> Running the build in mock_runner locally targeted for el7, el8, fc29 >>>>> and fc30 work just fine. >>>>> >>>>> Help please! >>>>> >>>>> [1] - https://gerrit.ovirt.org/#/c/107309/ >>>>> [2] - >>>>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >>>>> [3] - >>>>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >>>>> >>>>> >>>>> -- >>>>> Scott Dickerson >>>>> Senior Software Engineer >>>>> RHV-M Engineering - UX Team >>>>> Red Hat, Inc >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Infra mailing list -- infra(a)ovirt.org >>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... >>>>> >>>> >>>> >>>> -- >>>> Barak Korren >>>> RHV DevOps team , RHCE, RHCi >>>> Red Hat EMEA >>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>> >>>> >>>> >> >> -- >> Scott Dickerson >> Senior Software Engineer >> RHV-M Engineering - UX Team >> Red Hat, Inc >> > > > -- > Sandro Bonazzola > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://www.redhat.com/>*Red Hat respects your work life balance. > Therefore there is no need to answer this email out of your office hours. > <https://mojo.redhat.com/docs/DOC-1199578>* > > > -- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc

On 6 Mar 2020, at 13:41, Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; wrote: Any other suggestion? It’s still mostly failing, just see builds at https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/ > On 3 Mar 2020, at 21:56, Scott Dickerson <sdickers(a)redhat.com&gt; wrote: > > > > On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers(a)redhat.com&gt; wrote: > Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked. > > I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens. > > Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz" > > Regards, > Scott > > [1] - https://gerrit.ovirt.org/107367 > [2] - https://gerrit.ovirt.org/107381 > > Correction, the pre-seed patch is actually: https://gerrit.ovirt.org/107382 > > On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; wrote: > > >> On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: >> >> >> >> Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson <sdickers(a)redhat.com&gt; ha scritto: >> >> >> On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch(a)redhat.com&gt; wrote: >> >> >> On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; wrote: >> >> >>> On 3 Mar 2020, at 09:14, Barak Korren <bkorren(a)redhat.com&gt; wrote: >>> >>> It matters - because it would block sending the relevant builds to the CQ. >> >> yes >> >>> >>> One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server. >>> >>> A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case? >> >> good point, we can try that I guess. Weird thing is it succeeds some time. >> Scott, can you try that? >> >> >> >> @Sandro Bonazzola tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358 >> >> >> >> I was just watching the jenkins build on his patch and it failed with exactly the same error as before. >> >> FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-) >> >> Yes, still failing on >> error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". >> so even using a proxy is not helping at all. >> >> It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-( > > how to use NO_PROXY? We don’t really need nor want any proxy > > >> >> >> >> >> >> >>> >>> Why is this not failing in check-patch as well BTW? >> >> I think it doesn’t do any downloads, just checks the presence of package.json. >> >> Thanks, >> michal >> >>> >>> On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; wrote: >>> is this a better list? or no one cares? >>> >>>> On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; wrote: >>>> >>>> >>>> >>>>> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers(a)redhat.com&gt; wrote: >>>>> >>>>> Hi, >>>>> >>>>> On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log: >>>> >>>> it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. >>>> >>>>> >>>>> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har >>>>> [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har >>>>> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 >>>>> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... >>>>> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... >>>>> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". >>>>> >>>>> Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine. >>>>> >>>>> Help please! >>>>> >>>>> [1] - https://gerrit.ovirt.org/#/c/107309/ >>>>> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >>>>> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... >>>>> >>>>> -- >>>>> Scott Dickerson >>>>> Senior Software Engineer >>>>> RHV-M Engineering - UX Team >>>>> Red Hat, Inc >>>> >>> >>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... >>> >>> >>> -- >>> Barak Korren >>> RHV DevOps team , RHCE, RHCi >>> Red Hat EMEA >>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> >> >> >> -- >> Scott Dickerson >> Senior Software Engineer >> RHV-M Engineering - UX Team >> Red Hat, Inc >> >> >> -- >> Sandro Bonazzola >> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >> Red Hat EMEA >> sbonazzo(a)redhat.com >> >> Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. > > > > -- > Scott Dickerson > Senior Software Engineer > RHV-M Engineering - UX Team > Red Hat, Inc > > > -- > Scott Dickerson > Senior Software Engineer > RHV-M Engineering - UX Team > Red Hat, Inc

Looking at squid logs there have been no requests to theproxy for patternfly-2.59.2.tgz and the URL seems to work fine for the moment. This looks like an instability on the yarnpkg.com CDN and using the PHX proxy may indeed help stabilize things here. Seems like other people are facing similar issues as well: https://github.com/yarnpkg/yarn/issues/7521

Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go.

On Fri, Mar 6, 2020 at 2:59 PM Anton Marchukov <amarchuk(a)redhat.com&gt; wrote: > Adding Evgheni. > > > On 6 Mar 2020, at 13:41, Michal Skrivanek <michal.skrivanek(a)redhat.com&gt; > wrote: > > > > Any other suggestion? > > It’s still mostly failing, just see builds at > https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/ > > > > > >> On 3 Mar 2020, at 21:56, Scott Dickerson <sdickers(a)redhat.com&gt; wrote: > >> > >> > >> > >> On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers(a)redhat.com&gt; > wrote: > >> Note: I updated ovirt-engine-nodejs-modules with a new patch that did > not require any new yarn downloads [1]. This gets CI working for > ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a > pre-seed update. I created a new patch [2] to contain my pre-seed > requirements (and therefore new yarn downloads). CI needs to do a > successful 'ci build' on el7, el8, fc29 and fc30 before a handful of > patches can be unblocked. > >> > >> I looked into the http proxy and there is no way to turn them off from > being set when calling the automation scripts. I can adjust the scripts, > but since wget seems to work ok with the proxy environment vars, the > problem is with yarn. I'll try a few options, starting with telling yarn > to explicitly turn off the proxy and see what happens. > >> > >> Can someone look at the squid logs to see if anything is getting > through to the yarn repos? E.g. " > https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz > " > >> > >> Regards, > >> Scott > >> > >> [1] - https://gerrit.ovirt.org/107367 > >> [2] - https://gerrit.ovirt.org/107381 > >> > >> Correction, the pre-seed patch is actually: > https://gerrit.ovirt.org/107382 > >> > >> On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek < > michal.skrivanek(a)redhat.com&gt; wrote: > >> > >> > >>> On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; > wrote: > >>> > >>> > >>> > >>> Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson < > sdickers(a)redhat.com&gt; ha scritto: > >>> > >>> > >>> On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch(a)redhat.com&gt; > wrote: > >>> > >>> > >>> On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < > michal.skrivanek(a)redhat.com&gt; wrote: > >>> > >>> > >>>> On 3 Mar 2020, at 09:14, Barak Korren <bkorren(a)redhat.com&gt; wrote: > >>>> > >>>> It matters - because it would block sending the relevant builds to > the CQ. > >>> > >>> yes > >>> > >>>> > >>>> One thing that could make YARN fail in CI but not in local mock is > the fact the we have HTTP_PROXY defined in the CI environment and pointing > to a Squid server. > >>>> > >>>> A typical issue we see people having is when connecting to > `localhost` and ending up being blocked by the proxy. Please make sure the > NO_PROXY env var is set appropriately if that is the case? > >>> > >>> good point, we can try that I guess. Weird thing is it succeeds some > time. > >>> Scott, can you try that? > >>> > >>> > >>> > >>> @Sandro Bonazzola tries that by using a specific local proxy > solution: https://gerrit.ovirt.org/107358 > >>> > >>> > >>> > >>> I was just watching the jenkins build on his patch and it failed with > exactly the same error as before. > >>> > >>> FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) > only needs to do a total of 8 npm/yarnpkg repo downloads per > build-artifacts run. So it isn't exactly asking for a huge effort. ;-) > >>> > >>> Yes, still failing on > >>> error An unexpected error occurred: " > https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: > unexpected end of file". > >>> so even using a proxy is not helping at all. > >>> > >>> It appears to affect everyone using yarn and common solution I found > is just retry on failure till it works :-( > >> > >> how to use NO_PROXY? We don’t really need nor want any proxy > >> > >> > >>> > >>> > >>> > >>> > >>> > >>> > >>>> > >>>> Why is this not failing in check-patch as well BTW? > >>> > >>> I think it doesn’t do any downloads, just checks the presence of > package.json. > >>> > >>> Thanks, > >>> michal > >>> > >>>> > >>>> On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < > michal.skrivanek(a)redhat.com&gt; wrote: > >>>> is this a better list? or no one cares? > >>>> > >>>>> On 2 Mar 2020, at 09:40, Michal Skrivanek < > michal.skrivanek(a)redhat.com&gt; wrote: > >>>>> > >>>>> > >>>>> > >>>>>> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers(a)redhat.com&gt; > wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> On the merge phase on patch [1], both the el7 and fc30 build have > been failing. Examples are [2] and [3]. I'm guessing there are > environmental issues that I can't fix from the project's perspective. > Typical example of the error from the log: > >>>>> > >>>>> it seems to fail randomly, in run 93 el8 and fc30 fails while el7 > and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. > >>>>> > >>>>>> > >>>>>> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har > >>>>>> [2020-03-01T15:55:19.475Z] + > /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js > install --pure-lockfile --har > >>>>>> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 > >>>>>> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... > >>>>>> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... > >>>>>> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " > https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: > unexpected end of file". > >>>>>> > >>>>>> Running the build in mock_runner locally targeted for el7, el8, > fc29 and fc30 work just fine. > >>>>>> > >>>>>> Help please! > >>>>>> > >>>>>> [1] - https://gerrit.ovirt.org/#/c/107309/ > >>>>>> [2] - > https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... > >>>>>> [3] - > https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-... > >>>>>> > >>>>>> -- > >>>>>> Scott Dickerson > >>>>>> Senior Software Engineer > >>>>>> RHV-M Engineering - UX Team > >>>>>> Red Hat, Inc > >>>>> > >>>> > >>>> _______________________________________________ > >>>> Infra mailing list -- infra(a)ovirt.org > >>>> To unsubscribe send an email to infra-leave(a)ovirt.org > >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > >>>> oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > >>>> List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJ... > >>>> > >>>> > >>>> -- > >>>> Barak Korren > >>>> RHV DevOps team , RHCE, RHCi > >>>> Red Hat EMEA > >>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > >>> > >>> > >>> > >>> -- > >>> Scott Dickerson > >>> Senior Software Engineer > >>> RHV-M Engineering - UX Team > >>> Red Hat, Inc > >>> > >>> > >>> -- > >>> Sandro Bonazzola > >>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > >>> Red Hat EMEA > >>> sbonazzo(a)redhat.com > >>> > >>> Red Hat respects your work life balance. Therefore there is no need > to answer this email out of your office hours. > >> > >> > >> > >> -- > >> Scott Dickerson > >> Senior Software Engineer > >> RHV-M Engineering - UX Team > >> Red Hat, Inc > >> > >> > >> -- > >> Scott Dickerson > >> Senior Software Engineer > >> RHV-M Engineering - UX Team > >> Red Hat, Inc > > > > -- > Anton Marchukov > Associate Manager - RHV DevOps - Red Hat > > > > > > > -- Regards, Evgheni Dereveanchin

On 9 Mar 2020, at 09:09, Anton Marchukov <amarchuk(a)redhat.com&gt; wrote: Hello Scott. > On 6 Mar 2020, at 22:23, Scott Dickerson <sdickers(a)redhat.com&gt; wrote: > > Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go. > > > Caching on the proxy won't work since everything is https, unless squid is setup to intercept https traffic... Just to remind that in the past we have set up a Nexus server in oVirt PHX DC that is capable of caching nodejs artefacts. AFAIK it is not actively used, but does it makes sense to reconsider this decisions now? I think using Nexus is more clear that doing SSL MITM on existing squid (although this is also something doable).

-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat