On 2/11/20 1:36 PM, Marcin Sobczyk wrote:

On 2/11/20 1:16 PM, Barak Korren wrote:

...

On Tue, 11 Feb 2020 at 12:02, Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:

Hi,

agents used for CI runs for one of my patches [1] seem to be contaminated, i.e. our linters run complains about files like [2]:

[2020-02-11T09:16:00.381Z] ./.local/share/virtualenv/seed-v1/3.7/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel/bdist_wheel.py:135:80: E501 line too long (84 > 79 characters) [2020-02-11T09:16:00.381Z] raise ValueError('Unsupported compression: {}'.format(self.compression)) [2020-02-11T09:16:00.381Z] ^ [2020-02-11T09:16:00.381Z] ./.local/share/virtualenv/seed-v1/3.7/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel/bdist_wheel.py:145:80: E501 line too long (93 > 79 characters) [2020-02-11T09:16:00.381Z] if self.py_limited_api and not re.match(PY_LIMITED_API_PATTERN, self.py_limited_api):

tests are failing with [3]:

<https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-864>[2020-02-11T09:13:53.807Z] tox -e "tests,storage,lib,network,virt,gluster" <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-865>[2020-02-11T09:13:54.074Z] tests create: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/tests <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-866>[2020-02-11T09:13:54.336Z] ERROR: invocation failed (exit code 1), logfile: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/tests/log/tests-0.log

<https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-867>[2020-02-11T09:13:54.337Z] ================================== log start =================================== <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-868>[2020-02-11T09:13:54.337Z] ERROR:root:ImportError: cannot import name 'ensure_text'

The patch itself is minimal and the parent patch seemed to be fine [4].

I think the root cause for this is that inside mock $HOME == $PWD == where the repo is cloned.

Where we're seeing is that tox is trying to place the virtualenv it creates in $HOME/.local and then flake8 trying to scan everything under $PWD which happens to include the `.local` directory which ends up containing libraries that are not pep8 conforment....

Tox uses '.tox' directory to keep its stuff and we stick to specific version [5]. We also didn't have any changes around tox/CI recently - I still think it's a defunct agent.

[5] https://github.com/oVirt/vdsm/blob/9e1ea54bea2a3ea1b7d434617bd8445af4953f21/...

Ok, scratch that - I think it's due to new virtualenv version... let me try to fix the version to the older one for now.

...

To solve this we should either move the location where tox places virtualenvs (I suppose there is some env var that controls this), of make flak8 ignore the `.local` directoy.

Regards, Marcin

[1] https://gerrit.ovirt.org/#/c/106846/ [2] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_sta... [3] https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... [4] https://gerrit.ovirt.org/#/c/106590/ _______________________________________________ Infra mailing list -- infra@ovirt.org <mailto:infra@ovirt.org> To unsubscribe send an email to infra-leave@ovirt.org <mailto:infra-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/VJKVASATTJQ4ZW...

-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted>

On Tue, 11 Feb 2020 at 15:31, Marcin Sobczyk <msobczyk@redhat.com> wrote:

On 2/11/20 2:04 PM, Marcin Sobczyk wrote:

On 2/11/20 1:36 PM, Marcin Sobczyk wrote:

On 2/11/20 1:16 PM, Barak Korren wrote:

On Tue, 11 Feb 2020 at 12:02, Marcin Sobczyk <msobczyk@redhat.com> wrote:

...

Hi,

agents used for CI runs for one of my patches [1] seem to be contaminated, i.e. our linters run complains about files like [2]:

[2020-02-11T09:16:00.381Z] ./.local/share/virtualenv/seed-v1/3.7/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel/bdist_wheel.py:135:80: E501 line too long (84 > 79 characters) [2020-02-11T09:16:00.381Z] raise ValueError('Unsupported compression: {}'.format(self.compression)) [2020-02-11T09:16:00.381Z] ^ [2020-02-11T09:16:00.381Z] ./.local/share/virtualenv/seed-v1/3.7/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel/bdist_wheel.py:145:80: E501 line too long (93 > 79 characters) [2020-02-11T09:16:00.381Z] if self.py_limited_api and not re.match(PY_LIMITED_API_PATTERN, self.py_limited_api):

tests are failing with [3]:

<https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-864>[2020-02-11T09:13:53.807Z] tox -e "tests,storage,lib,network,virt,gluster" <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-865>[2020-02-11T09:13:54.074Z] tests create: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/tests <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-866>[2020-02-11T09:13:54.336Z] ERROR: invocation failed (exit code 1), logfile: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/tests/log/tests-0.log <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-867>[2020-02-11T09:13:54.337Z] ================================== log start =================================== <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/18071/pipeline/151#step-314-log-868>[2020-02-11T09:13:54.337Z] ERROR:root:ImportError: cannot import name 'ensure_text'

The patch itself is minimal and the parent patch seemed to be fine [4].

I think the root cause for this is that inside mock $HOME == $PWD == where the repo is cloned.

Where we're seeing is that tox is trying to place the virtualenv it creates in $HOME/.local and then flake8 trying to scan everything under $PWD which happens to include the `.local` directory which ends up containing libraries that are not pep8 conforment....

Tox uses '.tox' directory to keep its stuff and we stick to specific version [5]. We also didn't have any changes around tox/CI recently - I still think it's a defunct agent.

[5] https://github.com/oVirt/vdsm/blob/9e1ea54bea2a3ea1b7d434617bd8445af4953f21/...

Ok, scratch that - I think it's due to new virtualenv version... let me try to fix the version to the older one for now.

Yep, that did the trick: https://gerrit.ovirt.org/#/c/106877/ Sorry for the noise.

N/P - In our CI scripts we force the virtualenvs to go to a specific location which is also cached on the host, this prevents this kind of issue as well as make things run faster. Since we use pipenv, for us this was as simple as setting PIPENV_CACHE_DIR to a bind-mounted location. Btw leaving dirty nodes behind is close to impossible. `mock` takes care of 95% of things and `git clean -fdx` takes care of the rest. AFAIK the only way you can successfully leave files behind is by placing them under your git clone dire and including them in `.gitignore`.

Marcin

To solve this we should either move the location where tox places virtualenvs (I suppose there is some env var that controls this), of make flak8 ignore the `.local` directoy.

...

Regards, Marcin

[1] https://gerrit.ovirt.org/#/c/106846/ [2] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_sta... [3] https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... [4] https://gerrit.ovirt.org/#/c/106590/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/VJKVASATTJQ4ZW...

-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted