June 2018 - Infra - oVirt List Archives

[JIRA] (OVIRT-1965) Add failure details to CQ messages
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1965?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1965: -------------------------------- Epic Link: OVIRT-2185 (was: OVIRT-400) > Add failure details to CQ messages > ---------------------------------- > > Key: OVIRT-1965 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1965 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Change Queue > Reporter: Barak Korren > Assignee: infra > > In order to streamline the process for debugging CQ issues, we should have more failure details added to the messages it sends. Those details should include: > # The names of the OST suits that failed > # The OST tests that failed > # If it was a build failure rather then an OST failure > # If it was possible to detect - the particular change(s) that caused failure. > #if its possible to tell - if its an infra issue or a code regression > To implement this we would need at the very least a richer interface for passing information between the CQ tester and the and manager jobs - possibly some kind of a structured data file passed around. We may also need a richer interface to OST itself (to tell which test failed) and perhaps even to the build jobs (to tell why the build failed). -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1965) Add failure details to CQ messages
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1965?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1965: -------------------------------- Epic Link: OVIRT-2185 (was: OVIRT-400) > Add failure details to CQ messages > ---------------------------------- > > Key: OVIRT-1965 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1965 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: Change Queue > Reporter: Barak Korren > Assignee: infra > > In order to streamline the process for debugging CQ issues, we should have more failure details added to the messages it sends. Those details should include: > # The names of the OST suits that failed > # The OST tests that failed > # If it was a build failure rather then an OST failure > # If it was possible to detect - the particular change(s) that caused failure. > #if its possible to tell - if its an infra issue or a code regression > To implement this we would need at the very least a richer interface for passing information between the CQ tester and the and manager jobs - possibly some kind of a structured data file passed around. We may also need a richer interface to OST itself (to tell which test failed) and perhaps even to the build jobs (to tell why the build failed). -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1973) vdsm_master_check-patch-fc27 fail due to yum repo issues
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1973?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1973: -------------------------------- Issue Type: Outage (was: By-EMAIL) > vdsm_master_check-patch-fc27 fail due to yum repo issues > -------------------------------------------------------- > > Key: OVIRT-1973 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1973 > Project: oVirt - virtualization made easy > Issue Type: Outage > Components: CI client projects > Reporter: danken > Assignee: infra > > Once in a while, jobs are failing due to > [MIRROR] kmod-kvdo-6.1.1.24-1.fc27.x86_64.rpm: Downloading successful, > but checksum doesn't match. Calculated: > 54996f8e7877b533bf43825853cd3c7cfd0f7df5cb6a1876e50c9ade7829dd73(sha256) > Expected: 7a8294588764bd2c14adaa7d24218cce344930c1ce62a24b17208d1cd629c461(sha256) > for example > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc27-x86_64/2830/conso… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1973) vdsm_master_check-patch-fc27 fail due to yum repo issues
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1973?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1973: -------------------------------- Epic Link: (was: OVIRT-400) > vdsm_master_check-patch-fc27 fail due to yum repo issues > -------------------------------------------------------- > > Key: OVIRT-1973 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1973 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Components: CI client projects > Reporter: danken > Assignee: infra > > Once in a while, jobs are failing due to > [MIRROR] kmod-kvdo-6.1.1.24-1.fc27.x86_64.rpm: Downloading successful, > but checksum doesn't match. Calculated: > 54996f8e7877b533bf43825853cd3c7cfd0f7df5cb6a1876e50c9ade7829dd73(sha256) > Expected: 7a8294588764bd2c14adaa7d24218cce344930c1ce62a24b17208d1cd629c461(sha256) > for example > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc27-x86_64/2830/conso… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1973) vdsm_master_check-patch-fc27 fail due to yum repo issues
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1973?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1973: -------------------------------- Epic Link: (was: OVIRT-400) > vdsm_master_check-patch-fc27 fail due to yum repo issues > -------------------------------------------------------- > > Key: OVIRT-1973 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1973 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Components: CI client projects > Reporter: danken > Assignee: infra > > Once in a while, jobs are failing due to > [MIRROR] kmod-kvdo-6.1.1.24-1.fc27.x86_64.rpm: Downloading successful, > but checksum doesn't match. Calculated: > 54996f8e7877b533bf43825853cd3c7cfd0f7df5cb6a1876e50c9ade7829dd73(sha256) > Expected: 7a8294588764bd2c14adaa7d24218cce344930c1ce62a24b17208d1cd629c461(sha256) > for example > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc27-x86_64/2830/conso… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1983) Switch to using JJB >= 2.0.6
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1983?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1983: -------------------------------- Epic Link: (was: OVIRT-400) > Switch to using JJB >= 2.0.6 > ---------------------------- > > Key: OVIRT-1983 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1983 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: JJB > Reporter: Barak Korren > Assignee: infra > > Version 2.0.6 is the latest released version of JJB at the time of this writing. We've been using a very old development version of JJB (1.4.1.dev50) for a long time and its about time we upgraded. Other then that the post 2.0.0 has one major feature which is Jinja2 support which may be very useful for us. > In order to upgrade, we probably need to implement OVIRT-1466 first. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1983) Switch to using JJB >= 2.0.6
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1983?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1983: -------------------------------- Epic Link: (was: OVIRT-400) > Switch to using JJB >= 2.0.6 > ---------------------------- > > Key: OVIRT-1983 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1983 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: JJB > Reporter: Barak Korren > Assignee: infra > > Version 2.0.6 is the latest released version of JJB at the time of this writing. We've been using a very old development version of JJB (1.4.1.dev50) for a long time and its about time we upgraded. Other then that the post 2.0.0 has one major feature which is Jinja2 support which may be very useful for us. > In order to upgrade, we probably need to implement OVIRT-1466 first. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1984) Create "out-of-band" slave cleanup and setup jobs
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1984?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1984: -------------------------------- Epic Link: OVIRT-2178 (was: OVIRT-400) > Create "out-of-band" slave cleanup and setup jobs > ------------------------------------------------- > > Key: OVIRT-1984 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1984 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: Jenkins Slaves > Reporter: Barak Korren > Assignee: infra > > Right now, we run slave cleaup and setup steps as part or every single job we run. This has several shortcomings: > # It takes a long time from the point a user submitted a patch to the point his actual test or build code runs > # If slave setup or cleanup steps fail - they fail the whole job for the user > # If slave setup or cleanup steps fail - they can keep failing for many jobs until the CI team intervenes manually > # There is a "chicken and an egg" issue where some parts of the CI code have to run before the slave was properly cleaned up and configured. This makes if harder to add new slaves for the system. > Here is a suggested scheme to fix all this: > # Label all slaves that should be cleaned up automatically as 'cleanable'. This is mostly to prevent the jobs described here from operating on the master node. > # Have a "cleanup scheduler" job that finds all slaves labelled as "cleanable" but not as "dirty" or "clean", labels them as "dirty" and runs a cleanup job on them. > # Have a "cleanup" job that is triggered on particular slaves by the "cleanup scheduler" job, runs cleaup and setup steps on them and then labels them as "clean" and removes the "dirty" label. > # Have all other CI jobs only use slaves with the "clean" label. > Notes: > # The "dirty" label is there to make the "cleanup scheduler" job not trigger twice on the same slave before the"cleanup" job started cleaning it up. > # Since all slaves used by the real jobs will always be clean - there will no longer be a need to run cleanup steps in the real jobs, thus saving time. > # If cleanup steps fail - the cleanup job will fail and the slave will not be marked as "clean" so real jobs will never try to use it. > # To solve the "chicken and egg" issue, the cleanup job probably must be a FreeStyle jobs and all the cleanup and setup code must be embedded into it by JJB. This will probably require a newer version of JJB then what we have so setting OVIRT-1983 as a blocker. > # There is an issue of how to make CI for this - if cleanup and setup steps are removed from the normal STDCI jobs, they they will not be checked by the "check-patch" job of the "jenkins repo". Here is a suggested scheme to solve this: > ## Have a way to "loan" slaves from the production jenkins to other Jenkins instances - this could be done by having a job that starts up the Jenkins JNLP client and tells it to connect to another Jenkins master. > ## As part of the "check-patch" job for the 'jenkins' repo - start a Jenkins master in a container - attach some production slaves to it and have it run cleanup and setup steps on them -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1984) Create "out-of-band" slave cleanup and setup jobs
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1984?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1984: -------------------------------- Epic Link: OVIRT-2178 (was: OVIRT-400) > Create "out-of-band" slave cleanup and setup jobs > ------------------------------------------------- > > Key: OVIRT-1984 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1984 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: Jenkins Slaves > Reporter: Barak Korren > Assignee: infra > > Right now, we run slave cleaup and setup steps as part or every single job we run. This has several shortcomings: > # It takes a long time from the point a user submitted a patch to the point his actual test or build code runs > # If slave setup or cleanup steps fail - they fail the whole job for the user > # If slave setup or cleanup steps fail - they can keep failing for many jobs until the CI team intervenes manually > # There is a "chicken and an egg" issue where some parts of the CI code have to run before the slave was properly cleaned up and configured. This makes if harder to add new slaves for the system. > Here is a suggested scheme to fix all this: > # Label all slaves that should be cleaned up automatically as 'cleanable'. This is mostly to prevent the jobs described here from operating on the master node. > # Have a "cleanup scheduler" job that finds all slaves labelled as "cleanable" but not as "dirty" or "clean", labels them as "dirty" and runs a cleanup job on them. > # Have a "cleanup" job that is triggered on particular slaves by the "cleanup scheduler" job, runs cleaup and setup steps on them and then labels them as "clean" and removes the "dirty" label. > # Have all other CI jobs only use slaves with the "clean" label. > Notes: > # The "dirty" label is there to make the "cleanup scheduler" job not trigger twice on the same slave before the"cleanup" job started cleaning it up. > # Since all slaves used by the real jobs will always be clean - there will no longer be a need to run cleanup steps in the real jobs, thus saving time. > # If cleanup steps fail - the cleanup job will fail and the slave will not be marked as "clean" so real jobs will never try to use it. > # To solve the "chicken and egg" issue, the cleanup job probably must be a FreeStyle jobs and all the cleanup and setup code must be embedded into it by JJB. This will probably require a newer version of JJB then what we have so setting OVIRT-1983 as a blocker. > # There is an issue of how to make CI for this - if cleanup and setup steps are removed from the normal STDCI jobs, they they will not be checked by the "check-patch" job of the "jenkins repo". Here is a suggested scheme to solve this: > ## Have a way to "loan" slaves from the production jenkins to other Jenkins instances - this could be done by having a job that starts up the Jenkins JNLP client and tells it to connect to another Jenkins master. > ## As part of the "check-patch" job for the 'jenkins' repo - start a Jenkins master in a container - attach some production slaves to it and have it run cleanup and setup steps on them -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0

[JIRA] (OVIRT-1988) Add default behavior in secret_resolvers.py
by Barak Korren (oVirt JIRA) 12 Jun '18

12 Jun '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1988?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1988: -------------------------------- Labels: first_time_task (was: ) > Add default behavior in secret_resolvers.py > ------------------------------------------- > > Key: OVIRT-1988 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1988 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: mock_runner > Reporter: Daniel Belenky > Assignee: infra > Labels: first_time_task > > Currently, when users try to run CI locally (currently with mock_runner), > if their project has a secret dependency they have to either write a local > secrets file, or to temporary modify their environment.yaml. > The file format should have support for default behavior in case a secret > was not found. Maybe something similar to: > {code:java} > --- > - name: MY_SECRET > valueFrom: > secretKeyRef: > name: secret_name > key: key_from_secret > default: > from-env: ENV_VAR_NAME > # OR > value: default-value > {code} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100087)

1 0