Re: ovirt-engine-nodejs-modules build-artifacts failing on el7 and fc30
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ <https://gerrit.ovirt.org/#/c/107309/> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/88/pipeline> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/92/pipeline>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
It matters - because it would block sending the relevant builds to the CQ. One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server. A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case? Why is this not failing in check-patch as well BTW? On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json. Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ <https://gerrit.ovirt.org/#/c/107309/> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/88/pipeline> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/92/pipeline>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org <mailto:infra@ovirt.org> To unsubscribe send an email to infra-leave@ovirt.org <mailto:infra-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK... <https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXKD4JKSEHEQ6UUX6HXMD/>
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com/> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted>
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before. FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson <sdickers@redhat.com> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". so even using a proxy is not helping at all. It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com <mailto:sgratch@redhat.com>> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com <mailto:bkorren@redhat.com>> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <mailto:sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358 <https://gerrit.ovirt.org/107358>
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309 <https://gerrit.ovirt.org/107309>) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>: unexpected end of file". so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ <https://gerrit.ovirt.org/#/c/107309/> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/88/pipeline> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/92/pipeline>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org <mailto:infra@ovirt.org> To unsubscribe send an email to infra-leave@ovirt.org <mailto:infra-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK... <https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXKD4JKSEHEQ6UUX6HXMD/>
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com/> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <mailto:sbonazzo@redhat.com> <https://www.redhat.com/>Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>
Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked. I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens. Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz " Regards, Scott [1] - https://gerrit.ovirt.org/107367 [2] - https://gerrit.ovirt.org/107381 On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson < sdickers@redhat.com> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on
error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers@redhat.com> wrote:
Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked.
I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens.
Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz "
Regards, Scott
[1] - https://gerrit.ovirt.org/107367 [2] - https://gerrit.ovirt.org/107381
Correction, the pre-seed patch is actually: https://gerrit.ovirt.org/107382
On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson < sdickers@redhat.com> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on
error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
Any other suggestion? It’s still mostly failing, just see builds at https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/
On 3 Mar 2020, at 21:56, Scott Dickerson <sdickers@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> wrote: Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked.
I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens.
Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>"
Regards, Scott
[1] - https://gerrit.ovirt.org/107367 <https://gerrit.ovirt.org/107367> [2] - https://gerrit.ovirt.org/107381 <https://gerrit.ovirt.org/107381>
Correction, the pre-seed patch is actually: https://gerrit.ovirt.org/107382 <https://gerrit.ovirt.org/107382>
On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com <mailto:sgratch@redhat.com>> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com <mailto:bkorren@redhat.com>> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola <mailto:sbonazzo@redhat.com> tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358 <https://gerrit.ovirt.org/107358>
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309 <https://gerrit.ovirt.org/107309>) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>: unexpected end of file". so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote: is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com <mailto:sdickers@redhat.com>> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz <https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz>: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ <https://gerrit.ovirt.org/#/c/107309/> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/88/pipeline> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-modules_standard-on-merge/detail/ovirt-engine-nodejs-modules_standard-on-merge/92/pipeline>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org <mailto:infra@ovirt.org> To unsubscribe send an email to infra-leave@ovirt.org <mailto:infra-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK... <https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXKD4JKSEHEQ6UUX6HXMD/>
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com/> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <mailto:sbonazzo@redhat.com> <https://www.redhat.com/>Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
Adding Evgheni.
On 6 Mar 2020, at 13:41, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
Any other suggestion? It’s still mostly failing, just see builds at https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/
On 3 Mar 2020, at 21:56, Scott Dickerson <sdickers@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers@redhat.com> wrote: Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked.
I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens.
Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz"
Regards, Scott
[1] - https://gerrit.ovirt.org/107367 [2] - https://gerrit.ovirt.org/107381
Correction, the pre-seed patch is actually: https://gerrit.ovirt.org/107382
On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson <sdickers@redhat.com> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to `localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <michal.skrivanek@redhat.com> wrote: is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote:
Hi,
On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
[2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har [2020-03-01T15:55:19.475Z] yarn install v1.17.3 [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... [2020-03-01T15:55:19.475Z] error An unexpected error occurred: "https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file".
Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine.
Help please!
[1] - https://gerrit.ovirt.org/#/c/107309/ [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod...
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA sbonazzo@redhat.com
Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Looking at squid logs there have been no requests to theproxy for patternfly-2.59.2.tgz and the URL seems to work fine for the moment. This looks like an instability on the yarnpkg.com CDN and using the PHX proxy may indeed help stabilize things here. Seems like other people are facing similar issues as well: https://github.com/yarnpkg/yarn/issues/7521 Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go. On Fri, Mar 6, 2020 at 2:59 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Adding Evgheni.
On 6 Mar 2020, at 13:41, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
Any other suggestion? It’s still mostly failing, just see builds at https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/
On 3 Mar 2020, at 21:56, Scott Dickerson <sdickers@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers@redhat.com> wrote: Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked.
I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens.
Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz "
Regards, Scott
[1] - https://gerrit.ovirt.org/107367 [2] - https://gerrit.ovirt.org/107381
Correction, the pre-seed patch is actually: https://gerrit.ovirt.org/107382
On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson < sdickers@redhat.com> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is
the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to
`localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of
package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <
michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
On 2 Mar 2020, at 09:40, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote: > > Hi, > > On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log:
it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded.
> > [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har > [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har > [2020-03-01T15:55:19.475Z] yarn install v1.17.3 > [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... > [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... > [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". > > Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine. > > Help please! > > [1] - https://gerrit.ovirt.org/#/c/107309/ > [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... > [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... > > -- > Scott Dickerson > Senior Software Engineer > RHV-M Engineering - UX Team > Red Hat, Inc
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA sbonazzo@redhat.com
Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
-- Regards, Evgheni Dereveanchin
On Fri, Mar 6, 2020 at 9:12 AM Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Looking at squid logs there have been no requests to theproxy for patternfly-2.59.2.tgz and the URL seems to work fine for the moment. This looks like an instability on the yarnpkg.com CDN and using the PHX proxy may indeed help stabilize things here.
Seems like other people are facing similar issues as well: https://github.com/yarnpkg/yarn/issues/7521
I've integrated some of the suggestions in that issue's thread with [0].
Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go.
Caching on the proxy won't work since everything is https, unless squid is setup to intercept https traffic... One thought - is there any difference between [1] and [2] on jenkins? Runners, disk size, memory size? I ask since I had a "ci build" run successfully and then within a few minutes have the merge job fail over and over. [0] - https://gerrit.ovirt.org/#/c/107492/ [1] - https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-check-pat... [2] - https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/
On Fri, Mar 6, 2020 at 2:59 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Adding Evgheni.
On 6 Mar 2020, at 13:41, Michal Skrivanek <michal.skrivanek@redhat.com> wrote:
Any other suggestion? It’s still mostly failing, just see builds at https://jenkins.ovirt.org/job/ovirt-engine-nodejs-modules_standard-on-merge/
On 3 Mar 2020, at 21:56, Scott Dickerson <sdickers@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:51 PM Scott Dickerson <sdickers@redhat.com> wrote: Note: I updated ovirt-engine-nodejs-modules with a new patch that did not require any new yarn downloads [1]. This gets CI working for ovirt-engine-ui-extensions and ovirt-web-ui changes that do not require a pre-seed update. I created a new patch [2] to contain my pre-seed requirements (and therefore new yarn downloads). CI needs to do a successful 'ci build' on el7, el8, fc29 and fc30 before a handful of patches can be unblocked.
I looked into the http proxy and there is no way to turn them off from being set when calling the automation scripts. I can adjust the scripts, but since wget seems to work ok with the proxy environment vars, the problem is with yarn. I'll try a few options, starting with telling yarn to explicitly turn off the proxy and see what happens.
Can someone look at the squid logs to see if anything is getting through to the yarn repos? E.g. " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz "
Regards, Scott
[1] - https://gerrit.ovirt.org/107367 [2] - https://gerrit.ovirt.org/107381
Correction, the pre-seed patch is actually: https://gerrit.ovirt.org/107382
On Tue, Mar 3, 2020 at 10:20 AM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 15:18, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 3 mar 2020 alle ore 15:09 Scott Dickerson < sdickers@redhat.com> ha scritto:
On Tue, Mar 3, 2020 at 8:35 AM Sharon Gratch <sgratch@redhat.com> wrote:
On Tue, Mar 3, 2020 at 3:17 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 3 Mar 2020, at 09:14, Barak Korren <bkorren@redhat.com> wrote:
It matters - because it would block sending the relevant builds to the CQ.
yes
One thing that could make YARN fail in CI but not in local mock is
the fact the we have HTTP_PROXY defined in the CI environment and pointing to a Squid server.
A typical issue we see people having is when connecting to
`localhost` and ending up being blocked by the proxy. Please make sure the NO_PROXY env var is set appropriately if that is the case?
good point, we can try that I guess. Weird thing is it succeeds some time. Scott, can you try that?
@Sandro Bonazzola tries that by using a specific local proxy solution: https://gerrit.ovirt.org/107358
I was just watching the jenkins build on his patch and it failed with exactly the same error as before.
FWIW, the pre-seed patch in question (https://gerrit.ovirt.org/107309) only needs to do a total of 8 npm/yarnpkg repo downloads per build-artifacts run. So it isn't exactly asking for a huge effort. ;-)
Yes, still failing on error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". so even using a proxy is not helping at all.
It appears to affect everyone using yarn and common solution I found is just retry on failure till it works :-(
how to use NO_PROXY? We don’t really need nor want any proxy
Why is this not failing in check-patch as well BTW?
I think it doesn’t do any downloads, just checks the presence of
package.json.
Thanks, michal
On Tue, 3 Mar 2020 at 10:07, Michal Skrivanek <
michal.skrivanek@redhat.com> wrote:
is this a better list? or no one cares?
> On 2 Mar 2020, at 09:40, Michal Skrivanek < michal.skrivanek@redhat.com> wrote: > > > >> On 1 Mar 2020, at 17:54, Scott Dickerson <sdickers@redhat.com> wrote: >> >> Hi, >> >> On the merge phase on patch [1], both the el7 and fc30 build have been failing. Examples are [2] and [3]. I'm guessing there are environmental issues that I can't fix from the project's perspective. Typical example of the error from the log: > > it seems to fail randomly, in run 93 el8 and fc30 fails while el7 and fc29 succeeds, in run 94 it’s fc30 that’s failing and el8 succeeded. > >> >> [2020-03-01T15:55:19.475Z] + yarn install --pure-lockfile --har >> [2020-03-01T15:55:19.475Z] + /home/jenkins/workspace/ovirt-engine-nodejs-modules_standard-on-merge/ovirt-engine-nodejs-modules/yarn-1.17.3.js install --pure-lockfile --har >> [2020-03-01T15:55:19.475Z] yarn install v1.17.3 >> [2020-03-01T15:55:19.475Z] [1/5] Resolving packages... >> [2020-03-01T15:55:19.475Z] [2/5] Fetching packages... >> [2020-03-01T15:55:19.475Z] error An unexpected error occurred: " https://registry.yarnpkg.com/@patternfly/react-core/-/react-core-3.134.2.tgz: unexpected end of file". >> >> Running the build in mock_runner locally targeted for el7, el8, fc29 and fc30 work just fine. >> >> Help please! >> >> [1] - https://gerrit.ovirt.org/#/c/107309/ >> [2] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... >> [3] - https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-engine-nodejs-mod... >> >> -- >> Scott Dickerson >> Senior Software Engineer >> RHV-M Engineering - UX Team >> Red Hat, Inc >
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MJAIPCG6VJJAXK...
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA sbonazzo@redhat.com
Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
-- Regards, Evgheni Dereveanchin
-- Scott Dickerson Senior Software Engineer RHV-M Engineering - UX Team Red Hat, Inc
Hello Scott.
On 6 Mar 2020, at 22:23, Scott Dickerson <sdickers@redhat.com> wrote:
Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go.
Caching on the proxy won't work since everything is https, unless squid is setup to intercept https traffic...
Just to remind that in the past we have set up a Nexus server in oVirt PHX DC that is capable of caching nodejs artefacts. AFAIK it is not actively used, but does it makes sense to reconsider this decisions now? I think using Nexus is more clear that doing SSL MITM on existing squid (although this is also something doable). -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
On 9 Mar 2020, at 09:09, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello Scott.
On 6 Mar 2020, at 22:23, Scott Dickerson <sdickers@redhat.com> wrote:
Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go.
Caching on the proxy won't work since everything is https, unless squid is setup to intercept https traffic...
Just to remind that in the past we have set up a Nexus server in oVirt PHX DC that is capable of caching nodejs artefacts. AFAIK it is not actively used, but does it makes sense to reconsider this decisions now? I think using Nexus is more clear that doing SSL MITM on existing squid (although this is also something doable).
But we can’t easily just change the urls in upstream code unless that server is accessible externally
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
The nexus instance should be available for everyone: https://nexus.apps.ovirt.org/repository/npm/ Is it possible to set several repos in yarn so that it can access upstream directly if nexus is not reachable for some reason? Alternatively, can we just pass a custom repo in pur CI as an env var or just create it for a run that runs in our CI so that nexus is used only when a certain condition is met? A quick google search shows that yarn cannot handle multiple repos by default: https://github.com/yarnpkg/yarn/issues/547 I am not sure however if there are common workarounds for this in practise that could allow us to use nexus and fail over if it's not reachable. On Mon, Mar 9, 2020 at 10:12 AM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 9 Mar 2020, at 09:09, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello Scott.
On 6 Mar 2020, at 22:23, Scott Dickerson <sdickers@redhat.com> wrote:
Advice ranges from increasing yarn timeouts to decreasing the size of packages uploaded to their registry so caching stuff on the proxy and not relying on upstream seems like the way to go.
Caching on the proxy won't work since everything is https, unless squid is setup to intercept https traffic...
Just to remind that in the past we have set up a Nexus server in oVirt PHX DC that is capable of caching nodejs artefacts. AFAIK it is not actively used, but does it makes sense to reconsider this decisions now? I think using Nexus is more clear that doing SSL MITM on existing squid (although this is also something doable).
But we can’t easily just change the urls in upstream code unless that server is accessible externally
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
-- Regards, Evgheni Dereveanchin
participants (7)
-
Anton Marchukov -
Barak Korren -
Evgheni Dereveanchin -
Michal Skrivanek -
Sandro Bonazzola -
Scott Dickerson -
Sharon Gratch