- Infra - oVirt List Archives

[JIRA] (OVIRT-1983) Switch to using JJB >= 2.0.6
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1983?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1983: -------------------------------- Epic Link: (was: OVIRT-400) > Switch to using JJB >= 2.0.6 > ---------------------------- > > Key: OVIRT-1983 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1983 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: JJB > Reporter: Barak Korren > Assignee: infra > > Version 2.0.6 is the latest released version of JJB at the time of this writing. We've been using a very old development version of JJB (1.4.1.dev50) for a long time and its about time we upgraded. Other then that the post 2.0.0 has one major feature which is Jinja2 support which may be very useful for us. > In order to upgrade, we probably need to implement OVIRT-1466 first. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1983) Switch to using JJB >= 2.0.6
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1983?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1983: -------------------------------- Epic Link: (was: OVIRT-400) > Switch to using JJB >= 2.0.6 > ---------------------------- > > Key: OVIRT-1983 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1983 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: JJB > Reporter: Barak Korren > Assignee: infra > > Version 2.0.6 is the latest released version of JJB at the time of this writing. We've been using a very old development version of JJB (1.4.1.dev50) for a long time and its about time we upgraded. Other then that the post 2.0.0 has one major feature which is Jinja2 support which may be very useful for us. > In order to upgrade, we probably need to implement OVIRT-1466 first. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1984) Create "out-of-band" slave cleanup and setup jobs
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1984?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1984: -------------------------------- Epic Link: OVIRT-2178 (was: OVIRT-400) > Create "out-of-band" slave cleanup and setup jobs > ------------------------------------------------- > > Key: OVIRT-1984 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1984 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: Jenkins Slaves > Reporter: Barak Korren > Assignee: infra > > Right now, we run slave cleaup and setup steps as part or every single job we run. This has several shortcomings: > # It takes a long time from the point a user submitted a patch to the point his actual test or build code runs > # If slave setup or cleanup steps fail - they fail the whole job for the user > # If slave setup or cleanup steps fail - they can keep failing for many jobs until the CI team intervenes manually > # There is a "chicken and an egg" issue where some parts of the CI code have to run before the slave was properly cleaned up and configured. This makes if harder to add new slaves for the system. > Here is a suggested scheme to fix all this: > # Label all slaves that should be cleaned up automatically as 'cleanable'. This is mostly to prevent the jobs described here from operating on the master node. > # Have a "cleanup scheduler" job that finds all slaves labelled as "cleanable" but not as "dirty" or "clean", labels them as "dirty" and runs a cleanup job on them. > # Have a "cleanup" job that is triggered on particular slaves by the "cleanup scheduler" job, runs cleaup and setup steps on them and then labels them as "clean" and removes the "dirty" label. > # Have all other CI jobs only use slaves with the "clean" label. > Notes: > # The "dirty" label is there to make the "cleanup scheduler" job not trigger twice on the same slave before the"cleanup" job started cleaning it up. > # Since all slaves used by the real jobs will always be clean - there will no longer be a need to run cleanup steps in the real jobs, thus saving time. > # If cleanup steps fail - the cleanup job will fail and the slave will not be marked as "clean" so real jobs will never try to use it. > # To solve the "chicken and egg" issue, the cleanup job probably must be a FreeStyle jobs and all the cleanup and setup code must be embedded into it by JJB. This will probably require a newer version of JJB then what we have so setting OVIRT-1983 as a blocker. > # There is an issue of how to make CI for this - if cleanup and setup steps are removed from the normal STDCI jobs, they they will not be checked by the "check-patch" job of the "jenkins repo". Here is a suggested scheme to solve this: > ## Have a way to "loan" slaves from the production jenkins to other Jenkins instances - this could be done by having a job that starts up the Jenkins JNLP client and tells it to connect to another Jenkins master. > ## As part of the "check-patch" job for the 'jenkins' repo - start a Jenkins master in a container - attach some production slaves to it and have it run cleanup and setup steps on them -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1984) Create "out-of-band" slave cleanup and setup jobs
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1984?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1984: -------------------------------- Epic Link: OVIRT-2178 (was: OVIRT-400) > Create "out-of-band" slave cleanup and setup jobs > ------------------------------------------------- > > Key: OVIRT-1984 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1984 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: Jenkins Slaves > Reporter: Barak Korren > Assignee: infra > > Right now, we run slave cleaup and setup steps as part or every single job we run. This has several shortcomings: > # It takes a long time from the point a user submitted a patch to the point his actual test or build code runs > # If slave setup or cleanup steps fail - they fail the whole job for the user > # If slave setup or cleanup steps fail - they can keep failing for many jobs until the CI team intervenes manually > # There is a "chicken and an egg" issue where some parts of the CI code have to run before the slave was properly cleaned up and configured. This makes if harder to add new slaves for the system. > Here is a suggested scheme to fix all this: > # Label all slaves that should be cleaned up automatically as 'cleanable'. This is mostly to prevent the jobs described here from operating on the master node. > # Have a "cleanup scheduler" job that finds all slaves labelled as "cleanable" but not as "dirty" or "clean", labels them as "dirty" and runs a cleanup job on them. > # Have a "cleanup" job that is triggered on particular slaves by the "cleanup scheduler" job, runs cleaup and setup steps on them and then labels them as "clean" and removes the "dirty" label. > # Have all other CI jobs only use slaves with the "clean" label. > Notes: > # The "dirty" label is there to make the "cleanup scheduler" job not trigger twice on the same slave before the"cleanup" job started cleaning it up. > # Since all slaves used by the real jobs will always be clean - there will no longer be a need to run cleanup steps in the real jobs, thus saving time. > # If cleanup steps fail - the cleanup job will fail and the slave will not be marked as "clean" so real jobs will never try to use it. > # To solve the "chicken and egg" issue, the cleanup job probably must be a FreeStyle jobs and all the cleanup and setup code must be embedded into it by JJB. This will probably require a newer version of JJB then what we have so setting OVIRT-1983 as a blocker. > # There is an issue of how to make CI for this - if cleanup and setup steps are removed from the normal STDCI jobs, they they will not be checked by the "check-patch" job of the "jenkins repo". Here is a suggested scheme to solve this: > ## Have a way to "loan" slaves from the production jenkins to other Jenkins instances - this could be done by having a job that starts up the Jenkins JNLP client and tells it to connect to another Jenkins master. > ## As part of the "check-patch" job for the 'jenkins' repo - start a Jenkins master in a container - attach some production slaves to it and have it run cleanup and setup steps on them -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1988) Add default behavior in secret_resolvers.py
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1988?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1988: -------------------------------- Labels: first_time_task (was: ) > Add default behavior in secret_resolvers.py > ------------------------------------------- > > Key: OVIRT-1988 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1988 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: mock_runner > Reporter: Daniel Belenky > Assignee: infra > Labels: first_time_task > > Currently, when users try to run CI locally (currently with mock_runner), > if their project has a secret dependency they have to either write a local > secrets file, or to temporary modify their environment.yaml. > The file format should have support for default behavior in case a secret > was not found. Maybe something similar to: > {code:java} > --- > - name: MY_SECRET > valueFrom: > secretKeyRef: > name: secret_name > key: key_from_secret > default: > from-env: ENV_VAR_NAME > # OR > value: default-value > {code} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1988) Add default behavior in secret_resolvers.py
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1988?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1988: -------------------------------- Epic Link: (was: OVIRT-400) > Add default behavior in secret_resolvers.py > ------------------------------------------- > > Key: OVIRT-1988 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1988 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: mock_runner > Reporter: Daniel Belenky > Assignee: infra > > Currently, when users try to run CI locally (currently with mock_runner), > if their project has a secret dependency they have to either write a local > secrets file, or to temporary modify their environment.yaml. > The file format should have support for default behavior in case a secret > was not found. Maybe something similar to: > {code:java} > --- > - name: MY_SECRET > valueFrom: > secretKeyRef: > name: secret_name > key: key_from_secret > default: > from-env: ENV_VAR_NAME > # OR > value: default-value > {code} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1988) Add default behavior in secret_resolvers.py
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1988?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1988: -------------------------------- Epic Link: (was: OVIRT-400) > Add default behavior in secret_resolvers.py > ------------------------------------------- > > Key: OVIRT-1988 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1988 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: mock_runner > Reporter: Daniel Belenky > Assignee: infra > > Currently, when users try to run CI locally (currently with mock_runner), > if their project has a secret dependency they have to either write a local > secrets file, or to temporary modify their environment.yaml. > The file format should have support for default behavior in case a secret > was not found. Maybe something similar to: > {code:java} > --- > - name: MY_SECRET > valueFrom: > secretKeyRef: > name: secret_name > key: key_from_secret > default: > from-env: ENV_VAR_NAME > # OR > value: default-value > {code} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1991) jenkins triggers a build on a deleted branch
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1991?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1991: -------------------------------- Epic Link: (was: OVIRT-400) > jenkins triggers a build on a deleted branch > -------------------------------------------- > > Key: OVIRT-1991 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1991 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Standard CI (Pipelines) > Reporter: Roy Golan > Assignee: infra > Priority: Low > > I did some pruning, and removed old, unused branches, from > https://github.com/oVirt/ovirt-openshift-extensions. Now I suddenly see my > jenkins build is failing, because it triggered a build on the deleted > branch. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1991) jenkins triggers a build on a deleted branch
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1991?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1991: -------------------------------- Epic Link: (was: OVIRT-400) > jenkins triggers a build on a deleted branch > -------------------------------------------- > > Key: OVIRT-1991 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1991 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Standard CI (Pipelines) > Reporter: Roy Golan > Assignee: infra > Priority: Low > > I did some pruning, and removed old, unused branches, from > https://github.com/oVirt/ovirt-openshift-extensions. Now I suddenly see my > jenkins build is failing, because it triggered a build on the deleted > branch. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1996) Remove the 4.1 nightly publisher job
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1996?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1996: -------------------------------- Epic Link: OVIRT-2044 (was: OVIRT-400) > Remove the 4.1 nightly publisher job > ------------------------------------ > > Key: OVIRT-1996 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1996 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Publisher jobs > Reporter: Barak Korren > Assignee: infra > Labels: first_time_task > > It may be the case that we don't need the 4.1 publisher jobs any more. > Since those jobs are still in the old pre-CQ format where artifacts we collected by the job itself, their existence makes the CI system's maintenance harder... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0