Proposal for new commit msg design for engine commits
Hi, You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI. Let me explain: Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch. So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt' 2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt' Jenkins jobs will then be able to parse that data and trigger only relevant jobs for it. this can also allow us to add more jobs per patch, an option that is very problematic today considering the amount of patches coming in to engine. Once agreed on a format, we'll be able to add a git hook to verify the validity of the commit msg. (similar to bug-url). if we're not 100% sure that the tags will cover all corner cases and we feel like we need to run the code on all jobs, we can a nightly job to run all the remaining jobs (but at least it won't run on every patch/commit). [1] <core | restapi | tools | history | engine | userportal | webadmin>: thoughts? Eyal Edri.
I like the idea of having a label in the bottom part of the commit that is: METADATA: network which would be your second proposal. ----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 11:38:51 AM Subject: [Engine-devel] Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt' 2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Jenkins jobs will then be able to parse that data and trigger only relevant jobs for it. this can also allow us to add more jobs per patch, an option that is very problematic today considering the amount of patches coming in to engine.
Once agreed on a format, we'll be able to add a git hook to verify the validity of the commit msg. (similar to bug-url).
if we're not 100% sure that the tags will cover all corner cases and we feel like we need to run the code on all jobs, we can a nightly job to run all the remaining jobs (but at least it won't run on every patch/commit).
[1] <core | restapi | tools | history | engine | userportal | webadmin>:
thoughts?
Eyal Edri. _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
On 07/09/2013 12:41 PM, Antoni Segura Puimedon wrote:
I like the idea of having a label in the bottom part of the commit that is:
METADATA: network
which would be your second proposal.
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 11:38:51 AM Subject: [Engine-devel] Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt' 2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Jenkins jobs will then be able to parse that data and trigger only relevant jobs for it. this can also allow us to add more jobs per patch, an option that is very problematic today considering the amount of patches coming in to engine.
Once agreed on a format, we'll be able to add a git hook to verify the validity of the commit msg. (similar to bug-url).
if we're not 100% sure that the tags will cover all corner cases and we feel like we need to run the code on all jobs, we can a nightly job to run all the remaining jobs (but at least it won't run on every patch/commit).
[1] <core | restapi | tools | history | engine | userportal | webadmin>:
thoughts?
Eyal Edri. _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
+1 beside of letting CI know which tests to run (main goal) it will also help people understand the scope and effect of the change on a quick look. i think that we can do feature based (live-snapshot,upgrade,live-migration...) or area base tagging (virt,storage,network..), what do you think?
On Tue, Jul 09, 2013 at 05:41:45AM -0400, Antoni Segura Puimedon wrote:
I like the idea of having a label in the bottom part of the commit that is:
METADATA: network
This makes sense, but I find "METADATA" as an overly general term. Even something like Region_of_Interest: network is more humanly comprehensible. Another idea would be to add a gerrit-only set of flags, where the poster or reviewers have to tick which group of tests are most important to be run.
which would be your second proposal.
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 11:38:51 AM Subject: [Engine-devel] Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt' 2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Jenkins jobs will then be able to parse that data and trigger only relevant jobs for it. this can also allow us to add more jobs per patch, an option that is very problematic today considering the amount of patches coming in to engine.
Once agreed on a format, we'll be able to add a git hook to verify the validity of the commit msg. (similar to bug-url).
if we're not 100% sure that the tags will cover all corner cases and we feel like we need to run the code on all jobs, we can a nightly job to run all the remaining jobs (but at least it won't run on every patch/commit).
[1] <core | restapi | tools | history | engine | userportal | webadmin>:
thoughts?
Eyal Edri. _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
now that gerrit was upgraded to 2.6.1 this becomes much more easier with custom fields options.[1] [1] http://gerrit-documentation.googlecode.com/svn/ReleaseNotes/ReleaseNotes-2.6... "The patch set review screen can include radio buttons for custom labels if enabled by submit rules. ". ----- Original Message -----
From: "Antoni Segura Puimedon" <asegurap@redhat.com> To: "infra" <infra@ovirt.org> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, July 9, 2013 12:41:45 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
I like the idea of having a label in the bottom part of the commit that is:
METADATA: network
which would be your second proposal.
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 11:38:51 AM Subject: [Engine-devel] Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt' 2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Jenkins jobs will then be able to parse that data and trigger only relevant jobs for it. this can also allow us to add more jobs per patch, an option that is very problematic today considering the amount of patches coming in to engine.
Once agreed on a format, we'll be able to add a git hook to verify the validity of the commit msg. (similar to bug-url).
if we're not 100% sure that the tags will cover all corner cases and we feel like we need to run the code on all jobs, we can a nightly job to run all the remaining jobs (but at least it won't run on every patch/commit).
[1] <core | restapi | tools | history | engine | userportal | webadmin>:
thoughts?
Eyal Edri. _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having: CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz Is much better. However, I don't think that it is the responsibility of the committer. I suggested some time ago to have metadata information within each source. Each source should have metadata of: 1. Maintainer group. 2. Subsystem. 3. Any other relevant. This way you can act automatically using this information. Java: /* * $OVIRT_METADATA.COMPONENT=<string> */ or: // $OVIRT_METADATA.COMPONENT=<string> Shell/python: # $OVIRT_METADATA.COMPONENT=<string> Well, you get the idea... This signature will allow: a. find . -type f | xargs grep '\$OVIRT_METADATA.COMPONENT=mycomponent' b. Future automation within gerrit. Regards, Alon
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
However, I don't think that it is the responsibility of the committer.
I suggested some time ago to have metadata information within each source.
Each source should have metadata of:
1. Maintainer group.
Not sure if this is always relevant, what if i'm fixing some unit test that got broken? it's enough to know I'm a maintainer, no?
2. Subsystem. 3. Any other relevant.
This way you can act automatically using this information.
Java: /* * $OVIRT_METADATA.COMPONENT=<string> */ or: // $OVIRT_METADATA.COMPONENT=<string>
Shell/python: # $OVIRT_METADATA.COMPONENT=<string>
Well, you get the idea...
This signature will allow: a. find . -type f | xargs grep '\$OVIRT_METADATA.COMPONENT=mycomponent' b. Future automation within gerrit.
Regards, Alon _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the project but it lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve the ability to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the form of: (either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
However, I don't think that it is the responsibility of the committer.
I suggested some time ago to have metadata information within each source.
Each source should have metadata of:
1. Maintainer group.
Not sure if this is always relevant, what if i'm fixing some unit test that got broken? it's enough to know I'm a maintainer, no?
Drop the maintainer, it is a suggestion of extra metadata we can add to automatically CC relevant people. Add more metadata, so that every change in that file will trigger specific tests.
2. Subsystem. 3. Any other relevant.
This way you can act automatically using this information.
Java: /* * $OVIRT_METADATA.COMPONENT=<string> */ or: // $OVIRT_METADATA.COMPONENT=<string>
Shell/python: # $OVIRT_METADATA.COMPONENT=<string>
Well, you get the idea...
This signature will allow: a. find . -type f | xargs grep '\$OVIRT_METADATA.COMPONENT=mycomponent' b. Future automation within gerrit.
Regards, Alon _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the
lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve
to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
(either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable). - fabian
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the
lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve
to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
(either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i don't understand what you mean by "a testcase should know what it covers". of course it does. that's not the problem though. the problem is ovirt-engine has multiple components with him that apply to various fields, some patches are relevant to specific areas and not not. so a "test case" (that clones ovirt-engine repo) can't know which tests to run if he doesn't have that info somewhere in the patch. if we don't want info in the git commit msg, we can try using git notes [1] but i think that this meta-data isn't all about ci jobs, it's about additional info on what the patch touches/adds. it can also help qa testing later to pinpoint where the code was changed and perhaps enable the use of future automation scripts that can scan the commits and generate statics on coverage per subject/field. [1] https://www.kernel.org/pub/software/scm/git/docs/git-notes.html http://alblue.bandlem.com/2011/11/git-tip-of-week-git-notes.html
- fabian
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the
lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve
to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
(either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"? let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message". now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit). Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests. now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario? i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important. eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
On 07/10/2013 10:27 PM, Eyal Edri wrote:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the
lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve
to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
(either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
we could use gerrit labels per test topic?
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Wednesday, July 10, 2013 10:37:03 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/10/2013 10:27 PM, Eyal Edri wrote:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message ----- > From: "Eyal Edri" <eedri@redhat.com> > To: "engine-devel" <engine-devel@ovirt.org> > Cc: "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 12:38:51 PM > Subject: Proposal for new commit msg design for engine commits > > Hi, > > You all probably know and familiar with 'ovirt-engine' git hook for > commit > msg template [1]. > this helps understand the general area of the patch in the
> lacks additional info that might > be valuable for scaling automatic tests in Jenkins CI. > > Let me explain: > > Infra team is working hard on expanding oVirt CI infrastructure and > adding > more tests in jenkins (per commit/patch). > Adding important meta-data per patch can significatly improve
> to > run specific tests for each patch/commit, > and not waste valuable resources on Jenkins jobs that are not relevant to > the > code in the patch. > > So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
> (either) > 1. expanding current header template to include more data like 'network' > , > 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
> 2. adding a new label with relevant tags for the patch, called e.g > 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
we could use gerrit labels per test topic?
sure, if gerrit allows adding any metadata per commit that doesn't have to be in the commit msg and can be read via api per patch, that will work as well.
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
On 07/10/2013 10:50 PM, Eyal Edri wrote:
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Wednesday, July 10, 2013 10:37:03 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/10/2013 10:27 PM, Eyal Edri wrote:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message ----- > From: "Alon Bar-Lev" <alonbl@redhat.com> > To: "Eyal Edri" <eedri@redhat.com> > Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 3:33:57 PM > Subject: Re: [Engine-devel] Proposal for new commit msg design for engine > commits > > > > ----- Original Message ----- >> From: "Eyal Edri" <eedri@redhat.com> >> To: "engine-devel" <engine-devel@ovirt.org> >> Cc: "infra" <infra@ovirt.org> >> Sent: Tuesday, July 9, 2013 12:38:51 PM >> Subject: Proposal for new commit msg design for engine commits >> >> Hi, >> >> You all probably know and familiar with 'ovirt-engine' git hook for >> commit >> msg template [1]. >> this helps understand the general area of the patch in the
>> lacks additional info that might >> be valuable for scaling automatic tests in Jenkins CI. >> >> Let me explain: >> >> Infra team is working hard on expanding oVirt CI infrastructure and >> adding >> more tests in jenkins (per commit/patch). >> Adding important meta-data per patch can significatly improve
>> to >> run specific tests for each patch/commit, >> and not waste valuable resources on Jenkins jobs that are not relevant to >> the >> code in the patch. >> >> So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
>> (either) >> 1. expanding current header template to include more data like 'network' >> , >> 'setup', 'tools', 'virt' > > Please do not expand header, it is too short anyway. > >> 2. adding a new label with relevant tags for the patch, called e.g >> 'METADATA: network, rest, virt' > > Having: > > CI-Tests: xxx > CI-Tests: yyy > CI-Tests: zzz > > Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
we could use gerrit labels per test topic?
sure, if gerrit allows adding any metadata per commit that doesn't have to be in the commit msg and can be read via api per patch, that will work as well.
soon... (gerrit 2.6) though categories will probably be the same for all projects, so not sure its the best approach for something with more than a few. (could also be done via a review comment in gerrit i guess which is free text)
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Wednesday, July 10, 2013 10:37:03 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/10/2013 10:27 PM, Eyal Edri wrote:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message ----- > From: "Eyal Edri" <eedri@redhat.com> > To: "engine-devel" <engine-devel@ovirt.org> > Cc: "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 12:38:51 PM > Subject: Proposal for new commit msg design for engine commits > > Hi, > > You all probably know and familiar with 'ovirt-engine' git hook for > commit > msg template [1]. > this helps understand the general area of the patch in the
> lacks additional info that might > be valuable for scaling automatic tests in Jenkins CI. > > Let me explain: > > Infra team is working hard on expanding oVirt CI infrastructure and > adding > more tests in jenkins (per commit/patch). > Adding important meta-data per patch can significatly improve
> to > run specific tests for each patch/commit, > and not waste valuable resources on Jenkins jobs that are not relevant to > the > code in the patch. > > So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
> (either) > 1. expanding current header template to include more data like 'network' > , > 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
> 2. adding a new label with relevant tags for the patch, called e.g > 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
we could use gerrit labels per test topic?
I don't think it should be by patch bases, but common logic based of metadata, if we do this within source tree maintainer can choose which tests he wish to have by simply commit metadata into the source tree. Asking for a special test can always be done in jenkins post commit, or by some generic field in gerrit. However, the standard test suites is something that should be applied via policy of the component in question using approved metadata.
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
----- Original Message -----
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Wednesday, July 10, 2013 10:37:03 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/10/2013 10:27 PM, Eyal Edri wrote:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message ----- > From: "Alon Bar-Lev" <alonbl@redhat.com> > To: "Eyal Edri" <eedri@redhat.com> > Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 3:33:57 PM > Subject: Re: [Engine-devel] Proposal for new commit msg design for engine > commits > > > > ----- Original Message ----- >> From: "Eyal Edri" <eedri@redhat.com> >> To: "engine-devel" <engine-devel@ovirt.org> >> Cc: "infra" <infra@ovirt.org> >> Sent: Tuesday, July 9, 2013 12:38:51 PM >> Subject: Proposal for new commit msg design for engine commits >> >> Hi, >> >> You all probably know and familiar with 'ovirt-engine' git hook for >> commit >> msg template [1]. >> this helps understand the general area of the patch in the
>> lacks additional info that might >> be valuable for scaling automatic tests in Jenkins CI. >> >> Let me explain: >> >> Infra team is working hard on expanding oVirt CI infrastructure and >> adding >> more tests in jenkins (per commit/patch). >> Adding important meta-data per patch can significatly improve
>> to >> run specific tests for each patch/commit, >> and not waste valuable resources on Jenkins jobs that are not relevant to >> the >> code in the patch. >> >> So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
>> (either) >> 1. expanding current header template to include more data like 'network' >> , >> 'setup', 'tools', 'virt' > > Please do not expand header, it is too short anyway. > >> 2. adding a new label with relevant tags for the patch, called e.g >> 'METADATA: network, rest, virt' > > Having: > > CI-Tests: xxx > CI-Tests: yyy > CI-Tests: zzz > > Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
we could use gerrit labels per test topic?
I don't think it should be by patch bases, but common logic based of metadata, if we do this within source tree maintainer can choose which tests he wish to have by simply commit metadata into the source tree.
Asking for a special test can always be done in jenkins post commit, or by some generic field in gerrit.
However, the standard test suites is something that should be applied via policy of the component in question using approved metadata.
+1 It's up to the maintainers to decide which tests cover their code, not up to the committer.
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Cc: "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 12:38:51 PM Subject: Proposal for new commit msg design for engine commits
Hi,
You all probably know and familiar with 'ovirt-engine' git hook for commit msg template [1]. this helps understand the general area of the patch in the
lacks additional info that might be valuable for scaling automatic tests in Jenkins CI.
Let me explain:
Infra team is working hard on expanding oVirt CI infrastructure and adding more tests in jenkins (per commit/patch). Adding important meta-data per patch can significatly improve
to run specific tests for each patch/commit, and not waste valuable resources on Jenkins jobs that are not relevant to the code in the patch.
So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
(either) 1. expanding current header template to include more data like 'network' , 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
2. adding a new label with relevant tags for the patch, called e.g 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
Hey Eyal, Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run). But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean). For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
The idea is to use the path/filename and dependency tree informations to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass But surely labels or meta-data in the commit msg are quicker to implement. - fabian
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, July 11, 2013 11:41:24 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:33:57 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message ----- > From: "Eyal Edri" <eedri@redhat.com> > To: "engine-devel" <engine-devel@ovirt.org> > Cc: "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 12:38:51 PM > Subject: Proposal for new commit msg design for engine commits > > Hi, > > You all probably know and familiar with 'ovirt-engine' git hook for > commit > msg template [1]. > this helps understand the general area of the patch in the
> lacks additional info that might > be valuable for scaling automatic tests in Jenkins CI. > > Let me explain: > > Infra team is working hard on expanding oVirt CI infrastructure and > adding > more tests in jenkins (per commit/patch). > Adding important meta-data per patch can significatly improve
> to > run specific tests for each patch/commit, > and not waste valuable resources on Jenkins jobs that are not relevant to > the > code in the patch. > > So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
> (either) > 1. expanding current header template to include more data like 'network' > , > 'setup', 'tools', 'virt'
Please do not expand header, it is too short anyway.
> 2. adding a new label with relevant tags for the patch, called e.g > 'METADATA: network, rest, virt'
Having:
CI-Tests: xxx CI-Tests: yyy CI-Tests: zzz
Is much better.
I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO.
I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable).
i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario?
Hey Eyal,
Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run).
this process can be fully automatic via gerrit hooks & templates: typos or mistakes can be easily handles by gerrit hooks to help the committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook. so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux. so the only task a committer will have to do is to verify the default tags are relevant. i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific code to relevant topic, statistics.. etc..)
But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean).
For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important.
The idea is to use the path/filename and dependency tree information to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass
I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every new file/path/class name that won't break that magic detection. sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components. if something will help, it will be splitting up ovirt-engine into subject projects (different git) Eyal.
But surely labels or meta-data in the commit msg are quicker to implement.
- fabian
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
On 07/11/2013 11:57 AM, Eyal Edri wrote:
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, July 11, 2013 11:41:24 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:42:24 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message ----- > From: "Alon Bar-Lev" <alonbl@redhat.com> > To: "Eyal Edri" <eedri@redhat.com> > Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 3:33:57 PM > Subject: Re: [Engine-devel] Proposal for new commit msg design for engine > commits > > > > ----- Original Message ----- >> From: "Eyal Edri" <eedri@redhat.com> >> To: "engine-devel" <engine-devel@ovirt.org> >> Cc: "infra" <infra@ovirt.org> >> Sent: Tuesday, July 9, 2013 12:38:51 PM >> Subject: Proposal for new commit msg design for engine commits >> >> Hi, >> >> You all probably know and familiar with 'ovirt-engine' git hook for >> commit >> msg template [1]. >> this helps understand the general area of the patch in the
>> lacks additional info that might >> be valuable for scaling automatic tests in Jenkins CI. >> >> Let me explain: >> >> Infra team is working hard on expanding oVirt CI infrastructure and >> adding >> more tests in jenkins (per commit/patch). >> Adding important meta-data per patch can significatly improve
>> to >> run specific tests for each patch/commit, >> and not waste valuable resources on Jenkins jobs that are not relevant to >> the >> code in the patch. >> >> So the idea is to add/expand current metadata per patch, in the
----- Original Message ----- project but it the ability form of:
>> (either) >> 1. expanding current header template to include more data like 'network' >> , >> 'setup', 'tools', 'virt' > Please do not expand header, it is too short anyway. > >> 2. adding a new label with relevant tags for the patch, called e.g >> 'METADATA: network, rest, virt' > Having: > > CI-Tests: xxx > CI-Tests: yyy > CI-Tests: zzz > > Is much better. I'm not sure we should have CI-Test - as we might use this for something else besides CI. Region_of_Interest as Dan suggests sounds better IMHO. I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable). i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario? Hey Eyal,
Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run).
----- Original Message ----- this process can be fully automatic via gerrit hooks & templates:
typos or mistakes can be easily handles by gerrit hooks to help the committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook.
so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux.
so the only task a committer will have to do is to verify the default tags are relevant.
i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific code to relevant topic, statistics.. etc..)
But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean).
For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important. The idea is to use the path/filename and dependency tree information to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every new file/path/class name that won't break that magic detection.
sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components.
if something will help, it will be splitting up ovirt-engine into subject projects (different git)
Eyal.
I think some valid points were raised in this thread, and I feel we all agree regarding the need for such a mechanism. regarding mapping of different areas in the code using metadata, i think this approach worth trying, it'll increase ownership and area of responsibility within our code and hopefully provide us the functionality we are looking for. we can start doing the obvious mapping, after-which the responsibility of each team/maintainer to assign a file to a person and define the specific functional areas in it. Moran.
But surely labels or meta-data in the commit msg are quicker to implement.
- fabian
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
OK, Rome wasn't built in a day. To move things forward, I propose we'll just improve current commit header template to include more relevant code areas [1], and start looking into mapping all code to the relevant components (either via renaming folders, adding a metadata file under each folder mapping the files/classnames/directory names or using automated tools like sonar) [1] instead of <core | restapi | tools | history | engine | userportal | webadmin> change to something like <core | restapi | tools | userportal | webadmin | network | storage | virt | packaging> Eyal. ----- Original Message -----
From: "Moran Goldboim" <mgoldboi@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Sunday, July 14, 2013 6:07:05 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/11/2013 11:57 AM, Eyal Edri wrote:
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, July 11, 2013 11:41:24 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev:
----- Original Message ----- > From: "Yair Zaslavsky" <yzaslavs@redhat.com> > To: "Alon Bar-Lev" <alonbl@redhat.com> > Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 3:42:24 PM > Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits > > > ----- Original Message ----- >> From: "Alon Bar-Lev" <alonbl@redhat.com> >> To: "Eyal Edri" <eedri@redhat.com> >> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> >> Sent: Tuesday, July 9, 2013 3:33:57 PM >> Subject: Re: [Engine-devel] Proposal for new commit msg design for engine >> commits >> >> >> >> ----- Original Message ----- >>> From: "Eyal Edri" <eedri@redhat.com> >>> To: "engine-devel" <engine-devel@ovirt.org> >>> Cc: "infra" <infra@ovirt.org> >>> Sent: Tuesday, July 9, 2013 12:38:51 PM >>> Subject: Proposal for new commit msg design for engine commits >>> >>> Hi, >>> >>> You all probably know and familiar with 'ovirt-engine' git hook for >>> commit >>> msg template [1]. >>> this helps understand the general area of the patch in the project but it >>> lacks additional info that might >>> be valuable for scaling automatic tests in Jenkins CI. >>> >>> Let me explain: >>> >>> Infra team is working hard on expanding oVirt CI infrastructure and >>> adding >>> more tests in jenkins (per commit/patch). >>> Adding important meta-data per patch can significatly improve the ability >>> to >>> run specific tests for each patch/commit, >>> and not waste valuable resources on Jenkins jobs that are not relevant to >>> the >>> code in the patch. >>> >>> So the idea is to add/expand current metadata per patch, in the form of: >>> (either) >>> 1. expanding current header template to include more data like 'network' >>> , >>> 'setup', 'tools', 'virt' >> Please do not expand header, it is too short anyway. >> >>> 2. adding a new label with relevant tags for the patch, called e.g >>> 'METADATA: network, rest, virt' >> Having: >> >> CI-Tests: xxx >> CI-Tests: yyy >> CI-Tests: zzz >> >> Is much better. > I'm not sure we should have CI-Test - as we might use this for something else > besides CI. > Region_of_Interest as Dan suggests sounds better IMHO. I don't care how this is to be called. However, I do not think that commit message is the place for instructing CI to do anything. Commit message stays for good, it should contain information that is required a year from now. It has nothing to do with tests and such.
I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable). i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario? Hey Eyal,
Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run).
----- Original Message ----- this process can be fully automatic via gerrit hooks & templates:
typos or mistakes can be easily handles by gerrit hooks to help the committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook.
so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux.
so the only task a committer will have to do is to verify the default tags are relevant.
i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific code to relevant topic, statistics.. etc..)
But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean).
For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important. The idea is to use the path/filename and dependency tree information to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every new file/path/class name that won't break that magic detection.
sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components.
if something will help, it will be splitting up ovirt-engine into subject projects (different git)
Eyal.
I think some valid points were raised in this thread, and I feel we all agree regarding the need for such a mechanism. regarding mapping of different areas in the code using metadata, i think this approach worth trying, it'll increase ownership and area of responsibility within our code and hopefully provide us the functionality we are looking for. we can start doing the obvious mapping, after-which the responsibility of each team/maintainer to assign a file to a person and define the specific functional areas in it.
Moran.
But surely labels or meta-data in the commit msg are quicker to implement.
- fabian
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org> Cc: "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 8:34:28 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
OK, Rome wasn't built in a day.
To move things forward, I propose we'll just improve current commit header template to include more relevant code areas [1], and start looking into mapping all code to the relevant components (either via renaming folders, adding a metadata file under each folder mapping the files/classnames/directory names or using automated tools like sonar)
Again, and I am sorry, but I disagree of any relationship between commit message and CI. It will be simple to add metadata to sources, and have CI run tests based on actual source change thus probable impact, this way we won't be exposed to human errors, nor make commit message unusable for actual history. All we need is someone to take ownership of the task of adding metadata to source tree. As I proposed this can be either within every source using special signature, or can be in a directory at special file, for example .ovirt-metadata, and have the mapping between source component to relevant tests at a simple text file at source root. Regards, Alon Bar-Lev.
[1] instead of <core | restapi | tools | history | engine | userportal | webadmin> change to something like <core | restapi | tools | userportal | webadmin | network | storage | virt | packaging>
Eyal.
----- Original Message -----
From: "Moran Goldboim" <mgoldboi@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Sunday, July 14, 2013 6:07:05 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/11/2013 11:57 AM, Eyal Edri wrote:
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, July 11, 2013 11:41:24 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri:
----- Original Message -----
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Tuesday, July 9, 2013 3:54:06 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev: > > ----- Original Message ----- >> From: "Yair Zaslavsky" <yzaslavs@redhat.com> >> To: "Alon Bar-Lev" <alonbl@redhat.com> >> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" > <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> >> Sent: Tuesday, July 9, 2013 3:42:24 PM >> Subject: Re: [Engine-devel] Proposal for new commit msg design for > engine commits >> >> >> ----- Original Message ----- >>> From: "Alon Bar-Lev" <alonbl@redhat.com> >>> To: "Eyal Edri" <eedri@redhat.com> >>> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" > <infra@ovirt.org> >>> Sent: Tuesday, July 9, 2013 3:33:57 PM >>> Subject: Re: [Engine-devel] Proposal for new commit msg design for > engine >>> commits >>> >>> >>> >>> ----- Original Message ----- >>>> From: "Eyal Edri" <eedri@redhat.com> >>>> To: "engine-devel" <engine-devel@ovirt.org> >>>> Cc: "infra" <infra@ovirt.org> >>>> Sent: Tuesday, July 9, 2013 12:38:51 PM >>>> Subject: Proposal for new commit msg design for engine commits >>>> >>>> Hi, >>>> >>>> You all probably know and familiar with 'ovirt-engine' git hook > for >>>> commit >>>> msg template [1]. >>>> this helps understand the general area of the patch in the > project but it >>>> lacks additional info that might >>>> be valuable for scaling automatic tests in Jenkins CI. >>>> >>>> Let me explain: >>>> >>>> Infra team is working hard on expanding oVirt CI infrastructure > and >>>> adding >>>> more tests in jenkins (per commit/patch). >>>> Adding important meta-data per patch can significatly improve > the ability >>>> to >>>> run specific tests for each patch/commit, >>>> and not waste valuable resources on Jenkins jobs that are not > relevant to >>>> the >>>> code in the patch. >>>> >>>> So the idea is to add/expand current metadata per patch, in the > form of: >>>> (either) >>>> 1. expanding current header template to include more data like > 'network' >>>> , >>>> 'setup', 'tools', 'virt' >>> Please do not expand header, it is too short anyway. >>> >>>> 2. adding a new label with relevant tags for the patch, called > e.g >>>> 'METADATA: network, rest, virt' >>> Having: >>> >>> CI-Tests: xxx >>> CI-Tests: yyy >>> CI-Tests: zzz >>> >>> Is much better. >> I'm not sure we should have CI-Test - as we might use this for > something else >> besides CI. >> Region_of_Interest as Dan suggests sounds better IMHO. > I don't care how this is to be called. > However, I do not think that commit message is the place for > instructing CI to do anything. > Commit message stays for good, it should contain information that is > required a year from now. > It has nothing to do with tests and such. I agree with Alon here that the Ci informations don't belong in the commit msg. My opinion is that a testcase should know what it covers. This information from the testcase can then be used by any party to determin if the testcase should be run on a specific commit (which yields informations about the changed paths, files, owner, author, etc ... which might be valuable). i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario? Hey Eyal,
Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run).
----- Original Message ----- this process can be fully automatic via gerrit hooks & templates:
typos or mistakes can be easily handles by gerrit hooks to help the committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook.
so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux.
so the only task a committer will have to do is to verify the default tags are relevant.
i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific code to relevant topic, statistics.. etc..)
But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean).
For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important. The idea is to use the path/filename and dependency tree information to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every new file/path/class name that won't break that magic detection.
sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components.
if something will help, it will be splitting up ovirt-engine into subject projects (different git)
Eyal.
I think some valid points were raised in this thread, and I feel we all agree regarding the need for such a mechanism. regarding mapping of different areas in the code using metadata, i think this approach worth trying, it'll increase ownership and area of responsibility within our code and hopefully provide us the functionality we are looking for. we can start doing the obvious mapping, after-which the responsibility of each team/maintainer to assign a file to a person and define the specific functional areas in it.
Moran.
But surely labels or meta-data in the commit msg are quicker to implement.
- fabian
eyal.
- fabian
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the commit code. Nevertheless, i have no problems with your suggestions for metadata per directory to map all ovirt code. any suggestion how to push it forward? Eyal. ----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:34:13 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org> Cc: "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 8:34:28 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
OK, Rome wasn't built in a day.
To move things forward, I propose we'll just improve current commit header template to include more relevant code areas [1], and start looking into mapping all code to the relevant components (either via renaming folders, adding a metadata file under each folder mapping the files/classnames/directory names or using automated tools like sonar)
Again, and I am sorry, but I disagree of any relationship between commit message and CI.
It will be simple to add metadata to sources, and have CI run tests based on actual source change thus probable impact, this way we won't be exposed to human errors, nor make commit message unusable for actual history.
All we need is someone to take ownership of the task of adding metadata to source tree.
As I proposed this can be either within every source using special signature, or can be in a directory at special file, for example .ovirt-metadata, and have the mapping between source component to relevant tests at a simple text file at source root.
Regards, Alon Bar-Lev.
[1] instead of <core | restapi | tools | history | engine | userportal | webadmin> change to something like <core | restapi | tools | userportal | webadmin | network | storage | virt | packaging>
Eyal.
----- Original Message -----
From: "Moran Goldboim" <mgoldboi@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Sunday, July 14, 2013 6:07:05 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/11/2013 11:57 AM, Eyal Edri wrote:
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, July 11, 2013 11:41:24 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri:
----- Original Message ----- > From: "Fabian Deutsch" <fabiand@redhat.com> > To: "Alon Bar-Lev" <alonbl@redhat.com> > Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" > <infra@ovirt.org> > Sent: Tuesday, July 9, 2013 3:54:06 PM > Subject: Re: [Engine-devel] Proposal for new commit msg design for > engine > commits > > Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev: >> >> ----- Original Message ----- >>> From: "Yair Zaslavsky" <yzaslavs@redhat.com> >>> To: "Alon Bar-Lev" <alonbl@redhat.com> >>> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" >> <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> >>> Sent: Tuesday, July 9, 2013 3:42:24 PM >>> Subject: Re: [Engine-devel] Proposal for new commit msg design for >> engine commits >>> >>> >>> ----- Original Message ----- >>>> From: "Alon Bar-Lev" <alonbl@redhat.com> >>>> To: "Eyal Edri" <eedri@redhat.com> >>>> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" >> <infra@ovirt.org> >>>> Sent: Tuesday, July 9, 2013 3:33:57 PM >>>> Subject: Re: [Engine-devel] Proposal for new commit msg design >>>> for >> engine >>>> commits >>>> >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Eyal Edri" <eedri@redhat.com> >>>>> To: "engine-devel" <engine-devel@ovirt.org> >>>>> Cc: "infra" <infra@ovirt.org> >>>>> Sent: Tuesday, July 9, 2013 12:38:51 PM >>>>> Subject: Proposal for new commit msg design for engine commits >>>>> >>>>> Hi, >>>>> >>>>> You all probably know and familiar with 'ovirt-engine' git hook >> for >>>>> commit >>>>> msg template [1]. >>>>> this helps understand the general area of the patch in the >> project but it >>>>> lacks additional info that might >>>>> be valuable for scaling automatic tests in Jenkins CI. >>>>> >>>>> Let me explain: >>>>> >>>>> Infra team is working hard on expanding oVirt CI infrastructure >> and >>>>> adding >>>>> more tests in jenkins (per commit/patch). >>>>> Adding important meta-data per patch can significatly improve >> the ability >>>>> to >>>>> run specific tests for each patch/commit, >>>>> and not waste valuable resources on Jenkins jobs that are not >> relevant to >>>>> the >>>>> code in the patch. >>>>> >>>>> So the idea is to add/expand current metadata per patch, in the >> form of: >>>>> (either) >>>>> 1. expanding current header template to include more data like >> 'network' >>>>> , >>>>> 'setup', 'tools', 'virt' >>>> Please do not expand header, it is too short anyway. >>>> >>>>> 2. adding a new label with relevant tags for the patch, called >> e.g >>>>> 'METADATA: network, rest, virt' >>>> Having: >>>> >>>> CI-Tests: xxx >>>> CI-Tests: yyy >>>> CI-Tests: zzz >>>> >>>> Is much better. >>> I'm not sure we should have CI-Test - as we might use this for >> something else >>> besides CI. >>> Region_of_Interest as Dan suggests sounds better IMHO. >> I don't care how this is to be called. >> However, I do not think that commit message is the place for >> instructing CI to do anything. >> Commit message stays for good, it should contain information that >> is >> required a year from now. >> It has nothing to do with tests and such. > I agree with Alon here that the Ci informations don't belong in the > commit msg. > My opinion is that a testcase should know what it covers. This > information from the testcase can then be used by any party to > determin > if the testcase should be run on a specific commit (which yields > informations about the changed paths, files, owner, author, etc ... > which might be valuable). i think you're missing the point here. can you explain how do you propose a test case will know "what it covers"?
let's take an example: let's say a new commit comes from ovirt-engine: http://gerrit.ovirt.org/#/c/16668/ commit msg: "core: Use images instead of volumes at CDA message".
now you have 1000 test cases (could be system or functional test). (let's assume that your infra can't support running 1000 tests per patch/commit).
Some of these test suits checks network flow, some virt (migration/template for e.g), some host install, others storage flows and so on... ). you have one repo to clone (ovirt-engine, let's keep vdsm a side for a min), and to compile the project from for the tests.
now given this scenario, please explain how will you know which test from the 1000 you have you'll run on it. do you believe that according to the author/path/filename you'll know if that patch involves storage or virt scenario? Hey Eyal,
Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run).
----- Original Message ----- this process can be fully automatic via gerrit hooks & templates:
typos or mistakes can be easily handles by gerrit hooks to help the committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook.
so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux.
so the only task a committer will have to do is to verify the default tags are relevant.
i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific code to relevant topic, statistics.. etc..)
But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean).
For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
i don't think there's an alternative to a metadata to assist mapping the patch to a relevant "topic" in the code. whether it exists as a git note or a label in the commit, that's another matter and probably less important. The idea is to use the path/filename and dependency tree information to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every new file/path/class name that won't break that magic detection.
sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components.
if something will help, it will be splitting up ovirt-engine into subject projects (different git)
Eyal.
I think some valid points were raised in this thread, and I feel we all agree regarding the need for such a mechanism. regarding mapping of different areas in the code using metadata, i think this approach worth trying, it'll increase ownership and area of responsibility within our code and hopefully provide us the functionality we are looking for. we can start doing the obvious mapping, after-which the responsibility of each team/maintainer to assign a file to a person and define the specific functional areas in it.
Moran.
But surely labels or meta-data in the commit msg are quicker to implement.
- fabian
eyal.
> - fabian > > _______________________________________________ > Engine-devel mailing list > Engine-devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel >
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:41:56 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the commit code.
This commit template is mostly invalid, as touching more than one 'subsystem' is possible, and has not enough granularity. For example, database change should trigger what? Infra change should trigger what? A change of both user interface and command should trigger what? So you end up with: userportal: storage: core: db: some message Just to make who happy? And maybe there are 50 tests of network, and you need only 5 of them for the specific change, how do you mark it, so now a developer need to know any test? what if you add one tomorrow which is relevant to a similar change? how do you inform the developer that now he needs 6? Why should it be the developer responsibility and not the quality ensuring engineer responsibility to determine which tests should run and when? As far as this template was not actually used for anything but humans, it was not that important, but once you formalize it as an interface, I step forward and state that the subject line is not the right tool for the task at hand (or any for this matter). The fact that you have in each commit are the sources that are modified, all the other data is just plain noise. From the sources that are modified you should be able to derive a test plan with high chance that this test program will be correct. Human intervention should be possible by ordering special tests that are outside of the standard policy, for cases in which the standard policy of deriving tests from sources is too narrow. Regards, Alon
Nevertheless, i have no problems with your suggestions for metadata per directory to map all ovirt code. any suggestion how to push it forward?
Eyal.
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:34:13 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org> Cc: "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 8:34:28 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
OK, Rome wasn't built in a day.
To move things forward, I propose we'll just improve current commit header template to include more relevant code areas [1], and start looking into mapping all code to the relevant components (either via renaming folders, adding a metadata file under each folder mapping the files/classnames/directory names or using automated tools like sonar)
Again, and I am sorry, but I disagree of any relationship between commit message and CI.
It will be simple to add metadata to sources, and have CI run tests based on actual source change thus probable impact, this way we won't be exposed to human errors, nor make commit message unusable for actual history.
All we need is someone to take ownership of the task of adding metadata to source tree.
As I proposed this can be either within every source using special signature, or can be in a directory at special file, for example .ovirt-metadata, and have the mapping between source component to relevant tests at a simple text file at source root.
Regards, Alon Bar-Lev.
[1] instead of <core | restapi | tools | history | engine | userportal | webadmin> change to something like <core | restapi | tools | userportal | webadmin | network | storage | virt | packaging>
Eyal.
----- Original Message -----
From: "Moran Goldboim" <mgoldboi@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Sunday, July 14, 2013 6:07:05 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/11/2013 11:57 AM, Eyal Edri wrote:
From: "Fabian Deutsch" <fabiand@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, July 11, 2013 11:41:24 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri: > ----- Original Message ----- >> From: "Fabian Deutsch" <fabiand@redhat.com> >> To: "Alon Bar-Lev" <alonbl@redhat.com> >> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" >> <infra@ovirt.org> >> Sent: Tuesday, July 9, 2013 3:54:06 PM >> Subject: Re: [Engine-devel] Proposal for new commit msg design for >> engine >> commits >> >> Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev: >>> >>> ----- Original Message ----- >>>> From: "Yair Zaslavsky" <yzaslavs@redhat.com> >>>> To: "Alon Bar-Lev" <alonbl@redhat.com> >>>> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" >>> <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> >>>> Sent: Tuesday, July 9, 2013 3:42:24 PM >>>> Subject: Re: [Engine-devel] Proposal for new commit msg design >>>> for >>> engine commits >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Alon Bar-Lev" <alonbl@redhat.com> >>>>> To: "Eyal Edri" <eedri@redhat.com> >>>>> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" >>> <infra@ovirt.org> >>>>> Sent: Tuesday, July 9, 2013 3:33:57 PM >>>>> Subject: Re: [Engine-devel] Proposal for new commit msg design >>>>> for >>> engine >>>>> commits >>>>> >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Eyal Edri" <eedri@redhat.com> >>>>>> To: "engine-devel" <engine-devel@ovirt.org> >>>>>> Cc: "infra" <infra@ovirt.org> >>>>>> Sent: Tuesday, July 9, 2013 12:38:51 PM >>>>>> Subject: Proposal for new commit msg design for engine commits >>>>>> >>>>>> Hi, >>>>>> >>>>>> You all probably know and familiar with 'ovirt-engine' git >>>>>> hook >>> for >>>>>> commit >>>>>> msg template [1]. >>>>>> this helps understand the general area of the patch in the >>> project but it >>>>>> lacks additional info that might >>>>>> be valuable for scaling automatic tests in Jenkins CI. >>>>>> >>>>>> Let me explain: >>>>>> >>>>>> Infra team is working hard on expanding oVirt CI >>>>>> infrastructure >>> and >>>>>> adding >>>>>> more tests in jenkins (per commit/patch). >>>>>> Adding important meta-data per patch can significatly improve >>> the ability >>>>>> to >>>>>> run specific tests for each patch/commit, >>>>>> and not waste valuable resources on Jenkins jobs that are not >>> relevant to >>>>>> the >>>>>> code in the patch. >>>>>> >>>>>> So the idea is to add/expand current metadata per patch, in >>>>>> the >>> form of: >>>>>> (either) >>>>>> 1. expanding current header template to include more data >>>>>> like >>> 'network' >>>>>> , >>>>>> 'setup', 'tools', 'virt' >>>>> Please do not expand header, it is too short anyway. >>>>> >>>>>> 2. adding a new label with relevant tags for the patch, >>>>>> called >>> e.g >>>>>> 'METADATA: network, rest, virt' >>>>> Having: >>>>> >>>>> CI-Tests: xxx >>>>> CI-Tests: yyy >>>>> CI-Tests: zzz >>>>> >>>>> Is much better. >>>> I'm not sure we should have CI-Test - as we might use this for >>> something else >>>> besides CI. >>>> Region_of_Interest as Dan suggests sounds better IMHO. >>> I don't care how this is to be called. >>> However, I do not think that commit message is the place for >>> instructing CI to do anything. >>> Commit message stays for good, it should contain information that >>> is >>> required a year from now. >>> It has nothing to do with tests and such. >> I agree with Alon here that the Ci informations don't belong in >> the >> commit msg. >> My opinion is that a testcase should know what it covers. This >> information from the testcase can then be used by any party to >> determin >> if the testcase should be run on a specific commit (which yields >> informations about the changed paths, files, owner, author, etc >> ... >> which might be valuable). > i think you're missing the point here. > can you explain how do you propose a test case will know "what it > covers"? > > let's take an example: > let's say a new commit comes from ovirt-engine: > http://gerrit.ovirt.org/#/c/16668/ > commit msg: "core: Use images instead of volumes at CDA message". > > now you have 1000 test cases (could be system or functional test). > (let's assume that your infra can't support running 1000 tests per > patch/commit). > > Some of these test suits checks network flow, some virt > (migration/template > for e.g), some host install, others storage flows and so on... ). > you have one repo to clone (ovirt-engine, let's keep vdsm a side > for > a > min), and to compile the project from for the tests. > > now given this scenario, please explain how will you know which > test > from > the 1000 you have you'll run on it. > do you believe that according to the author/path/filename you'll > know > if > that patch involves storage or virt scenario? Hey Eyal,
Yes - I would at least give it a try to decide automagically what tests to run by looking at the change. The main motivation for this is to not add another things which the committer needs to take care of and IMO we humans tend to fail (at some point) on those boring tasks like adding correct metadata (let it be a typo or just not adding the correct testsuites/topis to be run).
----- Original Message ----- this process can be fully automatic via gerrit hooks & templates:
typos or mistakes can be easily handles by gerrit hooks to help the committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook.
so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux.
so the only task a committer will have to do is to verify the default tags are relevant.
i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific code to relevant topic, statistics.. etc..)
But let's get back to your example. Basically we can use the path and filename to determin what testsuite to run. Testsuites could be bound to paths and/or filenames and/or regexes being matched against the full filename. Another approach would be to use a java package dependency tree to determine which classes are directly and indirectly affected by a change. This information can then be used to also build a set of testsuites to be run. For example: World uses Ocean uses Wale uses Cell - if Wale changes, we'll surely want to run the testsuites assigned to the classes higher up in the dependency chain (World and Ocean).
For the concrete example above: Maybe there is a spell checker testcase which could be bound to the filename glob pattern *.properties.
> i don't think there's an alternative to a metadata to assist > mapping > the > patch to a relevant "topic" in the code. > whether it exists as a git note or a label in the commit, that's > another > matter and probably less important. The idea is to use the path/filename and dependency tree information to model these topics. Example: WaterTestsuite(Topic): regex_in_change: .*\.fish glob_in_change: src/classes/ocean/*.java path_in_change: src/classes/water.java change_affects_depency_of: WaterClass I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every new file/path/class name that won't break that magic detection.
sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components.
if something will help, it will be splitting up ovirt-engine into subject projects (different git)
Eyal.
I think some valid points were raised in this thread, and I feel we all agree regarding the need for such a mechanism. regarding mapping of different areas in the code using metadata, i think this approach worth trying, it'll increase ownership and area of responsibility within our code and hopefully provide us the functionality we are looking for. we can start doing the obvious mapping, after-which the responsibility of each team/maintainer to assign a file to a person and define the specific functional areas in it.
Moran.
But surely labels or meta-data in the commit msg are quicker to implement.
- fabian
> eyal. > >> - fabian >> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >>
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --mJnleTaxCN8S70NxSM9ruh6OHmeqScDoW Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 07/20/2013 08:53 PM, Alon Bar-Lev wrote:
=20 =20 ----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>= , "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:41:56 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for eng= ine commits
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the = commit code. =20 This commit template is mostly invalid, as touching more than one 'subs= ystem' is possible, and has not enough granularity.
I suggest using a tag at the end with some extra syntax, like: Components: core, storage, db Components: all Components: all, !core, !db
=20 For example, database change should trigger what? All the jobs that are tagged for that component (db upgrades I suppose). And if the changes affect storage components then also storage, if the changes affect others then those others too.
Infra change should trigger what? The same, all the jobs that are tagged as infra.
A change of both user interface and command should trigger what? All the jobs tagged by user interface and/or command.
So you end up with: =20 userportal: storage: core: db: some message
As I suggested before, I think it's better if you end with a commit message like: Some message Components: userportal, storage, core, db Actually it can be easily done with a tag in the gerrit comment instead of the commit message.
=20 Just to make who happy?
The developer, the qe, the ci and the infra people. This mechanism is to avoid running all the tests all the time. Of course there are some times when all the tests should be run to make sure nothing else changed, but most times you just need to run part of them to make sure you did not break something obvious.
=20 And maybe there are 50 tests of network, and you need only 5 of them fo= r the specific change, how do you mark it, so now a developer need to kno= w any test? what if you add one tomorrow which is relevant to a similar c= hange? how do you inform the developer that now he needs 6?
As I said before, what the developer specifies is not a list of tests, but a list of components, that qe has to map to different sets of tests that can change with time. So specifying webadmin will run all the tests in that group, that might be only one, or 100, and might be increasing/decreasing with time transparently for the developer. Adding a new component is not common and there's no need to do it so frequently.=
=20 Why should it be the developer responsibility and not the quality ensur= ing engineer responsibility to determine which tests should run and when?=
Of course it's the responsibility of the qe engineer to determine when and which tests should be run. But this is meant to be a new tool for the developer not a substitute for the full qe tests, so the developer can easily make sure that he's changes do not break anything obvious before starting the real tests (that will take more time and resources). The developer just adds some metadata so the qe engineer can decide which tests to run per patch, so it's on qe's hand in the end to decide if ignore or not the metadata and which tests to run.
=20 As far as this template was not actually used for anything but humans, = it was not that important, but once you formalize it as an interface, I s= tep forward and state that the subject line is not the right tool for the= task at hand (or any for this matter).
I agree with that, I think that it should be a tag similar to Change-Id, at the end of the commit message.
=20 The fact that you have in each commit are the sources that are modified= , all the other data is just plain noise. From the sources that are modif= ied you should be able to derive a test plan with high chance that this t= est program will be correct. Human intervention should be possible by ord= ering special tests that are outside of the standard policy, for cases in= which the standard policy of deriving tests from sources is too narrow.
That's just not true. The sources are complicated enough to make two changes in the same file to affect different components. Any reused code is prone to affect multiple components, making it really hard to determine by which changed files which tests to run. And if you go down to the function/class level it's even harder to decide and to maintain. And of course it's not human error free, as the metadata in the files/directories is defined and maintained by a human. And in my opinion is a lot harder to implement and maintain, and a lot less agile, and does not get rid of the human factor.
=20 Regards, Alon =20 =20
Nevertheless, i have no problems with your suggestions for metadata pe=
r
directory to map all ovirt code. any suggestion how to push it forward?
Eyal.
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org= , "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:34:13 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for en= gine commits
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org> Cc: "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 8:34:28 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for e= ngine commits
OK, Rome wasn't built in a day.
To move things forward, I propose we'll just improve current commit header template to inclu= de more relevant code areas [1], and start looking into mapping all code to the relevant components (either via renaming folders, adding a metadata file under each fold= er mapping the files/classnames/directory names or using automated tool= s like sonar)
Again, and I am sorry, but I disagree of any relationship between com= mit message and CI.
It will be simple to add metadata to sources, and have CI run tests b= ased on actual source change thus probable impact, this way we won't be expos= ed to human errors, nor make commit message unusable for actual history.
All we need is someone to take ownership of the task of adding metada= ta to source tree.
As I proposed this can be either within every source using special signature, or can be in a directory at special file, for example .ovirt-metadata= , and have the mapping between source component to relevant tests at a simp= le text file at source root.
Regards, Alon Bar-Lev.
[1] instead of <core | restapi | tools | history | eng=
ine |
userportal | webadmin> change to something like <core | restapi | tools | userportal | webadmin | network | storage | virt | packaging>
Eyal.
----- Original Message -----
From: "Moran Goldboim" <mgoldboi@redhat.com> To: "Eyal Edri" <eedri@redhat.com> Cc: "Fabian Deutsch" <fabiand@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Sunday, July 14, 2013 6:07:05 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/11/2013 11:57 AM, Eyal Edri wrote:
----- Original Message ----- > From: "Fabian Deutsch" <fabiand@redhat.com> > To: "Eyal Edri" <eedri@redhat.com> > Cc: "Alon Bar-Lev" <alonbl@redhat.com>, "engine-devel" > <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> > Sent: Thursday, July 11, 2013 11:41:24 AM > Subject: Re: [Engine-devel] Proposal for new commit msg design fo=
r
> engine > commits > > Am Mittwoch, den 10.07.2013, 15:27 -0400 schrieb Eyal Edri: >> ----- Original Message ----- >>> From: "Fabian Deutsch" <fabiand@redhat.com> >>> To: "Alon Bar-Lev" <alonbl@redhat.com> >>> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" >>> <infra@ovirt.org> >>> Sent: Tuesday, July 9, 2013 3:54:06 PM >>> Subject: Re: [Engine-devel] Proposal for new commit msg design = for >>> engine >>> commits >>> >>> Am Dienstag, den 09.07.2013, 08:49 -0400 schrieb Alon Bar-Lev: >>>> >>>> ----- Original Message ----- >>>>> From: "Yair Zaslavsky" <yzaslavs@redhat.com> >>>>> To: "Alon Bar-Lev" <alonbl@redhat.com> >>>>> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" >>>> <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> >>>>> Sent: Tuesday, July 9, 2013 3:42:24 PM >>>>> Subject: Re: [Engine-devel] Proposal for new commit msg desig= n >>>>> for >>>> engine commits >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Alon Bar-Lev" <alonbl@redhat.com> >>>>>> To: "Eyal Edri" <eedri@redhat.com> >>>>>> Cc: "engine-devel" <engine-devel@ovirt.org>, "infra" >>>> <infra@ovirt.org> >>>>>> Sent: Tuesday, July 9, 2013 3:33:57 PM >>>>>> Subject: Re: [Engine-devel] Proposal for new commit msg desi= gn >>>>>> for >>>> engine >>>>>> commits >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Eyal Edri" <eedri@redhat.com> >>>>>>> To: "engine-devel" <engine-devel@ovirt.org> >>>>>>> Cc: "infra" <infra@ovirt.org> >>>>>>> Sent: Tuesday, July 9, 2013 12:38:51 PM >>>>>>> Subject: Proposal for new commit msg design for engine comm= its >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> You all probably know and familiar with 'ovirt-engine' git >>>>>>> hook >>>> for >>>>>>> commit >>>>>>> msg template [1]. >>>>>>> this helps understand the general area of the patch in the >>>> project but it >>>>>>> lacks additional info that might >>>>>>> be valuable for scaling automatic tests in Jenkins CI. >>>>>>> >>>>>>> Let me explain: >>>>>>> >>>>>>> Infra team is working hard on expanding oVirt CI >>>>>>> infrastructure >>>> and >>>>>>> adding >>>>>>> more tests in jenkins (per commit/patch). >>>>>>> Adding important meta-data per patch can significatly impro= ve >>>> the ability >>>>>>> to >>>>>>> run specific tests for each patch/commit, >>>>>>> and not waste valuable resources on Jenkins jobs that are n= ot >>>> relevant to >>>>>>> the >>>>>>> code in the patch. >>>>>>> >>>>>>> So the idea is to add/expand current metadata per patch, in=
>>>>>>> the >>>> form of: >>>>>>> (either) >>>>>>> 1. expanding current header template to include more data=
>>>>>>> like >>>> 'network' >>>>>>> , >>>>>>> 'setup', 'tools', 'virt' >>>>>> Please do not expand header, it is too short anyway. >>>>>> >>>>>>> 2. adding a new label with relevant tags for the patch, >>>>>>> called >>>> e.g >>>>>>> 'METADATA: network, rest, virt' >>>>>> Having: >>>>>> >>>>>> CI-Tests: xxx >>>>>> CI-Tests: yyy >>>>>> CI-Tests: zzz >>>>>> >>>>>> Is much better. >>>>> I'm not sure we should have CI-Test - as we might use this fo= r >>>> something else >>>>> besides CI. >>>>> Region_of_Interest as Dan suggests sounds better IMHO. >>>> I don't care how this is to be called. >>>> However, I do not think that commit message is the place for >>>> instructing CI to do anything. >>>> Commit message stays for good, it should contain information t= hat >>>> is >>>> required a year from now. >>>> It has nothing to do with tests and such. >>> I agree with Alon here that the Ci informations don't belong in=
>>> the >>> commit msg. >>> My opinion is that a testcase should know what it covers. This >>> information from the testcase can then be used by any party to >>> determin >>> if the testcase should be run on a specific commit (which yield= s >>> informations about the changed paths, files, owner, author, etc=
>>> ... >>> which might be valuable). >> i think you're missing the point here. >> can you explain how do you propose a test case will know "what i= t >> covers"? >> >> let's take an example: >> let's say a new commit comes from ovirt-engine: >> http://gerrit.ovirt.org/#/c/16668/ >> commit msg: "core: Use images instead of volumes at CDA message"= =2E >> >> now you have 1000 test cases (could be system or functional test= ). >> (let's assume that your infra can't support running 1000 tests p= er >> patch/commit). >> >> Some of these test suits checks network flow, some virt >> (migration/template >> for e.g), some host install, others storage flows and so on... )= =2E >> you have one repo to clone (ovirt-engine, let's keep vdsm a side=
>> for >> a >> min), and to compile the project from for the tests. >> >> now given this scenario, please explain how will you know which >> test >> from >> the 1000 you have you'll run on it. >> do you believe that according to the author/path/filename you'll=
>> know >> if >> that patch involves storage or virt scenario? > Hey Eyal, > > Yes - I would at least give it a try to decide automagically what=
> tests > to run by looking at the change. > The main motivation for this is to not add another things which t= he > committer needs to take care of and IMO we humans tend to fail (a= t > some > point) on those boring tasks like adding correct metadata (let it= be > a > typo or just not adding the correct testsuites/topis to be run). this process can be fully automatic via gerrit hooks & templates:
typos or mistakes can be easily handles by gerrit hooks to help th= e committer fix the tags. as mentions previously, this logic can be done by the project maintainer and enforced by a template or hook.
so for example - if someone writes a patch with patch header "webadmin:...." , then the tags he'll get to choose from are only relevant to ui/ux.=
so the only task a committer will have to do is to verify the defa=
ult
tags are relevant.
i don't believe this is too much to ask for, considering the huge benefit that we'll get (stable code, less bugs, less ci breakage, mapping of specific cod= e to relevant topic, statistics.. etc..)
> But let's get back to your example. > Basically we can use the path and filename to determin what > testsuite > to > run. > Testsuites could be bound to paths and/or filenames and/or regexe= s > being > matched against the full filename. > Another approach would be to use a java package dependency tree t= o > determine which classes are directly and indirectly affected by a=
> change. This information can then be used to also build a set of > testsuites to be run. For example: > World uses Ocean uses Wale uses Cell - if Wale changes, we'll sur= ely > want to run the testsuites assigned to the classes higher up in t= he > dependency chain (World and Ocean). > > For the concrete example above: Maybe there is a spell checker > testcase > which could be bound to the filename glob pattern *.properties. > >> i don't think there's an alternative to a metadata to assist >> mapping >> the >> patch to a relevant "topic" in the code. >> whether it exists as a git note or a label in the commit, that's=
>> another >> matter and probably less important. > The idea is to use the path/filename and dependency tree informat= ion > to > model these topics. Example: > WaterTestsuite(Topic): > regex_in_change: .*\.fish > glob_in_change: src/classes/ocean/*.java > path_in_change: src/classes/water.java > change_affects_depency_of: WaterClass I'm not familiar that much with the names of the classes and filenames, but it sounds to me like an error prone process and very complex to start going through all the classes and file names and mapping them to a certain project/component. sounds like we'll have to enforce a naming convention for every ne= w file/path/class name that won't break that magic detection.
sure there are exceptions that will work probably, like "anything under packaging/, should trigger the 'engine-setup' or 'engine-upgrade' tests, but imo, it is not so easy with other components.
if something will help, it will be splitting up ovirt-engine into subject projects (different git)
Eyal.
I think some valid points were raised in this thread, and I feel we= all agree regarding the need for such a mechanism. regarding mapping of different areas in the code using metadata, i think this approach worth trying, it'll increase ownership and area of responsibility within our code and hopefully provide us the functionality we are looking for. we can start doing the obvious mapping, after-which the responsibil= ity of each team/maintainer to assign a file to a person and define the=
specific functional areas in it.
Moran.
> But surely labels or meta-data in the commit msg are quicker to > implement. > > - fabian > >> eyal. >> >>> - fabian >>> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>> > > _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20
--=20 David Caro Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purky=C5=88ova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605 --mJnleTaxCN8S70NxSM9ruh6OHmeqScDoW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) iQEcBAEBAgAGBQJSFxpfAAoJEEBxx+HSYmnD1rkH/0gE9/aZRADQPB4TMvDTywl0 JSAcdc8T34Mvdv64GnlK2YRIDleKbGxFYzsuYAicfpts8C1ES+B58+t7jFboMGZP +vYkwSYCz1v/FWsgSnKLTwvHTNdnCD0vawZcVzM/G5cbIplY9IzO6jTnFGaO9KtW 9Uj5pRgBEGoXT8EXoM8iGcnPj5St4MxYiz+ZsNXNkD6ZAzNsOC2MvhVuQh01r6ld snkGzyu4C4l9ENlWWAMrk9wgdcCeK0ct3wQ7nHY+U6drOBDBjVGcPUV97TY02BJh Oxx9fNgyZj1PgU4rUcmNqR76qpMpfieSlMDbor8Dy8f/N6qTYGG+0B7kKuVoK+A= =uv5b -----END PGP SIGNATURE----- --mJnleTaxCN8S70NxSM9ruh6OHmeqScDoW--
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 11:16:31 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/20/2013 08:53 PM, Alon Bar-Lev wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:41:56 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the commit code.
This commit template is mostly invalid, as touching more than one 'subsystem' is possible, and has not enough granularity.
I suggest using a tag at the end with some extra syntax, like: Components: core, storage, db Components: all Components: all, !core, !db
For example, database change should trigger what?
All the jobs that are tagged for that component (db upgrades I suppose). And if the changes affect storage components then also storage, if the changes affect others then those others too.
Infra change should trigger what? The same, all the jobs that are tagged as infra.
A change of both user interface and command should trigger what? All the jobs tagged by user interface and/or command.
So you end up with:
userportal: storage: core: db: some message
As I suggested before, I think it's better if you end with a commit message like:
Some message
Components: userportal, storage, core, db
Actually it can be easily done with a tag in the gerrit comment instead of the commit message.
Just to make who happy?
The developer, the qe, the ci and the infra people. This mechanism is to avoid running all the tests all the time. Of course there are some times when all the tests should be run to make sure nothing else changed, but most times you just need to run part of them to make sure you did not break something obvious.
And maybe there are 50 tests of network, and you need only 5 of them for the specific change, how do you mark it, so now a developer need to know any test? what if you add one tomorrow which is relevant to a similar change? how do you inform the developer that now he needs 6?
As I said before, what the developer specifies is not a list of tests, but a list of components, that qe has to map to different sets of tests that can change with time. So specifying webadmin will run all the tests in that group, that might be only one, or 100, and might be increasing/decreasing with time transparently for the developer. Adding a new component is not common and there's no need to do it so frequently.
Why should it be the developer responsibility and not the quality ensuring engineer responsibility to determine which tests should run and when?
Of course it's the responsibility of the qe engineer to determine when and which tests should be run. But this is meant to be a new tool for the developer not a substitute for the full qe tests, so the developer can easily make sure that he's changes do not break anything obvious before starting the real tests (that will take more time and resources). The developer just adds some metadata so the qe engineer can decide which tests to run per patch, so it's on qe's hand in the end to decide if ignore or not the metadata and which tests to run.
As far as this template was not actually used for anything but humans, it was not that important, but once you formalize it as an interface, I step forward and state that the subject line is not the right tool for the task at hand (or any for this matter).
I agree with that, I think that it should be a tag similar to Change-Id, at the end of the commit message.
The fact that you have in each commit are the sources that are modified, all the other data is just plain noise. From the sources that are modified you should be able to derive a test plan with high chance that this test program will be correct. Human intervention should be possible by ordering special tests that are outside of the standard policy, for cases in which the standard policy of deriving tests from sources is too narrow.
That's just not true. The sources are complicated enough to make two changes in the same file to affect different components. Any reused code is prone to affect multiple components, making it really hard to determine by which changed files which tests to run. And if you go down to the function/class level it's even harder to decide and to maintain. And of course it's not human error free, as the metadata in the files/directories is defined and maintained by a human. And in my opinion is a lot harder to implement and maintain, and a lot less agile, and does not get rid of the human factor.
Once again... 1. Commit messages are not the place to specify metadata to interact with automation, it is a record for future reviewer. 2. The metadata within sources are the mean to automate the list of tests related to a specific source without human interaction on each commit. 3. If there is doubt from list of tests run them all, this is simple rule for automation. 4. The metadata within files are not the only way to order tests, one can do this manually via jenkins or any other mean as one can now. 5. The metadata within files will help us to achieve other targets, such as automatic CC maintainers, verify that +2/-2 are set by authorized maintainer of component etc... 6. Gerrit 2.6 supports labels[1] which are actually what you are trying to achieve using the commit message because of lack of other solutions. Until we have metadata within source, and we don't as we void discuss this for long time, and try to find manual workarounds and solutions. Add label for each test, this will allow ordering tests via the gerrit web interface post submit. After people start to do this over and over and over and over they will appreciate the need to add metadata into the source tree. In future, if this works out we can help gerrit to improve by enhancing the labels into free text/combo box etc... Or maybe try to do this now if you like to help out gerrit to improve... Regards, Alon Bar-Lev [1] http://gerrit-documentation.googlecode.com/svn/Documentation/2.6/config-labe...
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 10:45:37 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 11:16:31 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/20/2013 08:53 PM, Alon Bar-Lev wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:41:56 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the commit code.
This commit template is mostly invalid, as touching more than one 'subsystem' is possible, and has not enough granularity.
I suggest using a tag at the end with some extra syntax, like: Components: core, storage, db Components: all Components: all, !core, !db
For example, database change should trigger what?
All the jobs that are tagged for that component (db upgrades I suppose). And if the changes affect storage components then also storage, if the changes affect others then those others too.
Infra change should trigger what? The same, all the jobs that are tagged as infra.
A change of both user interface and command should trigger what? All the jobs tagged by user interface and/or command.
So you end up with:
userportal: storage: core: db: some message
As I suggested before, I think it's better if you end with a commit message like:
Some message
Components: userportal, storage, core, db
Actually it can be easily done with a tag in the gerrit comment instead of the commit message.
Just to make who happy?
The developer, the qe, the ci and the infra people. This mechanism is to avoid running all the tests all the time. Of course there are some times when all the tests should be run to make sure nothing else changed, but most times you just need to run part of them to make sure you did not break something obvious.
And maybe there are 50 tests of network, and you need only 5 of them for the specific change, how do you mark it, so now a developer need to know any test? what if you add one tomorrow which is relevant to a similar change? how do you inform the developer that now he needs 6?
As I said before, what the developer specifies is not a list of tests, but a list of components, that qe has to map to different sets of tests that can change with time. So specifying webadmin will run all the tests in that group, that might be only one, or 100, and might be increasing/decreasing with time transparently for the developer. Adding a new component is not common and there's no need to do it so frequently.
Why should it be the developer responsibility and not the quality ensuring engineer responsibility to determine which tests should run and when?
Of course it's the responsibility of the qe engineer to determine when and which tests should be run. But this is meant to be a new tool for the developer not a substitute for the full qe tests, so the developer can easily make sure that he's changes do not break anything obvious before starting the real tests (that will take more time and resources). The developer just adds some metadata so the qe engineer can decide which tests to run per patch, so it's on qe's hand in the end to decide if ignore or not the metadata and which tests to run.
As far as this template was not actually used for anything but humans, it was not that important, but once you formalize it as an interface, I step forward and state that the subject line is not the right tool for the task at hand (or any for this matter).
I agree with that, I think that it should be a tag similar to Change-Id, at the end of the commit message.
The fact that you have in each commit are the sources that are modified, all the other data is just plain noise. From the sources that are modified you should be able to derive a test plan with high chance that this test program will be correct. Human intervention should be possible by ordering special tests that are outside of the standard policy, for cases in which the standard policy of deriving tests from sources is too narrow.
That's just not true. The sources are complicated enough to make two changes in the same file to affect different components. Any reused code is prone to affect multiple components, making it really hard to determine by which changed files which tests to run. And if you go down to the function/class level it's even harder to decide and to maintain. And of course it's not human error free, as the metadata in the files/directories is defined and maintained by a human. And in my opinion is a lot harder to implement and maintain, and a lot less agile, and does not get rid of the human factor.
Once again...
1. Commit messages are not the place to specify metadata to interact with automation, it is a record for future reviewer.
Agree, that's why I suggested triggering it from a comment message. But that will require the developer another step after pushing.
2. The metadata within sources are the mean to automate the list of tests related to a specific source without human interaction on each commit.
The drawbacks are: - It needs a lot of maintenance - It require very modular code and - Locks the developer on which tests he want to run
3. If there is doubt from list of tests run them all, this is simple rule for automation.
Of course, but if there is too much doubt all that maintenance is useless.
4. The metadata within files are not the only way to order tests, one can do this manually via jenkins or any other mean as one can now.
As with any other solution.
5. The metadata within files will help us to achieve other targets, such as automatic CC maintainers, verify that +2/-2 are set by authorized maintainer of component etc...
That's true, but I think that maybe putting metadata in each file is overkilling. One file in the root of the repo with a few lines should be enough for that afaik.
6. Gerrit 2.6 supports labels[1] which are actually what you are trying to achieve using the commit message because of lack of other solutions.
They are not fit for that, at least yet.
Until we have metadata within source, and we don't as we void discuss this for long time, and try to find manual workarounds and solutions.
Or until we have tags in the messages, for the same reasons. Don't forget that metadata within source is also a 'manual workaround or solution', as someone has to maintain all the metadata in all the files/directories (maybe for that purpose will be better to have just one file with all that metadata, depending on which level of granularity we need to map all the files).
Add label for each test, this will allow ordering tests via the gerrit web interface post submit.
Adding a comment will too.
After people start to do this over and over and over and over they will appreciate the need to add metadata into the source tree.
I think that having to maintain metadata in all the files is way more annoying, but yes, as you say adding metadata to each file gives a lot more information, but only if it's up to date.
In future, if this works out we can help gerrit to improve by enhancing the labels into free text/combo box etc... Or maybe try to do this now if you like to help out gerrit to improve...
That will make labels fit our purpose, and avoid having to add tags, metadata and all that, but it's not done yet. I'll gladly add that feature, but I'm not familiar with java or gerrit code, so I might not be the best person to do it, I'm sure that it will take a lot more time for me than the other options (Eyal, any comment?). If you are fluent with java and you want to help you are very welcome :) David
----- Original Message -----
From: "David Caro Estevez" <dcaroest@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 1:00:38 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 10:45:37 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 11:16:31 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/20/2013 08:53 PM, Alon Bar-Lev wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:41:56 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the commit code.
This commit template is mostly invalid, as touching more than one 'subsystem' is possible, and has not enough granularity.
I suggest using a tag at the end with some extra syntax, like: Components: core, storage, db Components: all Components: all, !core, !db
For example, database change should trigger what?
All the jobs that are tagged for that component (db upgrades I suppose). And if the changes affect storage components then also storage, if the changes affect others then those others too.
Infra change should trigger what? The same, all the jobs that are tagged as infra.
A change of both user interface and command should trigger what? All the jobs tagged by user interface and/or command.
So you end up with:
userportal: storage: core: db: some message
As I suggested before, I think it's better if you end with a commit message like:
Some message
Components: userportal, storage, core, db
Actually it can be easily done with a tag in the gerrit comment instead of the commit message.
Just to make who happy?
The developer, the qe, the ci and the infra people. This mechanism is to avoid running all the tests all the time. Of course there are some times when all the tests should be run to make sure nothing else changed, but most times you just need to run part of them to make sure you did not break something obvious.
And maybe there are 50 tests of network, and you need only 5 of them for the specific change, how do you mark it, so now a developer need to know any test? what if you add one tomorrow which is relevant to a similar change? how do you inform the developer that now he needs 6?
As I said before, what the developer specifies is not a list of tests, but a list of components, that qe has to map to different sets of tests that can change with time. So specifying webadmin will run all the tests in that group, that might be only one, or 100, and might be increasing/decreasing with time transparently for the developer. Adding a new component is not common and there's no need to do it so frequently.
Why should it be the developer responsibility and not the quality ensuring engineer responsibility to determine which tests should run and when?
Of course it's the responsibility of the qe engineer to determine when and which tests should be run. But this is meant to be a new tool for the developer not a substitute for the full qe tests, so the developer can easily make sure that he's changes do not break anything obvious before starting the real tests (that will take more time and resources). The developer just adds some metadata so the qe engineer can decide which tests to run per patch, so it's on qe's hand in the end to decide if ignore or not the metadata and which tests to run.
As far as this template was not actually used for anything but humans, it was not that important, but once you formalize it as an interface, I step forward and state that the subject line is not the right tool for the task at hand (or any for this matter).
I agree with that, I think that it should be a tag similar to Change-Id, at the end of the commit message.
The fact that you have in each commit are the sources that are modified, all the other data is just plain noise. From the sources that are modified you should be able to derive a test plan with high chance that this test program will be correct. Human intervention should be possible by ordering special tests that are outside of the standard policy, for cases in which the standard policy of deriving tests from sources is too narrow.
That's just not true. The sources are complicated enough to make two changes in the same file to affect different components. Any reused code is prone to affect multiple components, making it really hard to determine by which changed files which tests to run. And if you go down to the function/class level it's even harder to decide and to maintain. And of course it's not human error free, as the metadata in the files/directories is defined and maintained by a human. And in my opinion is a lot harder to implement and maintain, and a lot less agile, and does not get rid of the human factor.
Once again...
1. Commit messages are not the place to specify metadata to interact with automation, it is a record for future reviewer.
Agree, that's why I suggested triggering it from a comment message. But that will require the developer another step after pushing.
2. The metadata within sources are the mean to automate the list of tests related to a specific source without human interaction on each commit.
The drawbacks are: - It needs a lot of maintenance
One time maintenance compared to per patch. Easier to enforce project policy, than relay on developer's policy.
- It require very modular code and
Right, we require this anyway.
- Locks the developer on which tests he want to run
As I wrote, it locks nothing, it is the baseline.
3. If there is doubt from list of tests run them all, this is simple rule for automation.
Of course, but if there is too much doubt all that maintenance is useless.
So you suggest we run partial? based on whose decision? can we trust him?
4. The metadata within files are not the only way to order tests, one can do this manually via jenkins or any other mean as one can now.
As with any other solution.
Right, because of that I do not think this is emergency. Just have jenkins jobs and people can order tests based on gerrit urls...
5. The metadata within files will help us to achieve other targets, such as automatic CC maintainers, verify that +2/-2 are set by authorized maintainer of component etc...
That's true, but I think that maybe putting metadata in each file is overkilling. One file in the root of the repo with a few lines should be enough for that afaik.
The metadata model I suggested was within each source and within directory. The collection is metadata of source + recursive meta data of directories. At file put signature. At directory put a file, such as .ovirt.metadata with same fields.
6. Gerrit 2.6 supports labels[1] which are actually what you are trying to achieve using the commit message because of lack of other solutions.
They are not fit for that, at least yet.
So maybe better if we can help gerrit to improve[1]? [1] http://code.google.com/p/gerrit/issues/detail?id=2085
Until we have metadata within source, and we don't as we void discuss this for long time, and try to find manual workarounds and solutions.
Or until we have tags in the messages, for the same reasons. Don't forget that metadata within source is also a 'manual workaround or solution', as someone has to maintain all the metadata in all the files/directories (maybe for that purpose will be better to have just one file with all that metadata, depending on which level of granularity we need to map all the files).
I think best is to spread metadata within the entire tree, as then maintainer of file control the metadata, if file is copied/duplicated/moved/renamed it keeps the metadata.
Add label for each test, this will allow ordering tests via the gerrit web interface post submit.
Adding a comment will too.
So why relay on commit message, you can just ask people to comment: OVIRT_ORDER_TEST: test1
After people start to do this over and over and over and over they will appreciate the need to add metadata into the source tree.
I think that having to maintain metadata in all the files is way more annoying, but yes, as you say adding metadata to each file gives a lot more information, but only if it's up to date.
Just like claiming that the source is out of date. If there is a mistake in metadata the maintainer will detect that when metadata is used.
In future, if this works out we can help gerrit to improve by enhancing the labels into free text/combo box etc... Or maybe try to do this now if you like to help out gerrit to improve...
That will make labels fit our purpose, and avoid having to add tags, metadata and all that, but it's not done yet. I'll gladly add that feature, but I'm not familiar with java or gerrit code, so I might not be the best person to do it, I'm sure that it will take a lot more time for me than the other options (Eyal, any comment?). If you are fluent with java and you want to help you are very welcome :)
You can help in making the metadata into the tree. This will be a great step forward. Thanks, Alon
On Fri 23 Aug 2013 12:19:01 PM CEST, Alon Bar-Lev wrote:
----- Original Message -----
From: "David Caro Estevez" <dcaroest@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 1:00:38 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "Alon Bar-Lev" <alonbl@redhat.com> To: "David Caro" <dcaroest@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 10:45:37 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "Eyal Edri" <eedri@redhat.com>, "engine-devel" <engine-devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Friday, August 23, 2013 11:16:31 AM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
On 07/20/2013 08:53 PM, Alon Bar-Lev wrote:
----- Original Message -----
From: "Eyal Edri" <eedri@redhat.com> To: "Alon Bar-Lev" <alonbl@redhat.com> Cc: "infra" <infra@ovirt.org>, "engine-devel" <engine-devel@ovirt.org>, "Fabian Deutsch" <fabiand@redhat.com> Sent: Saturday, July 20, 2013 9:41:56 PM Subject: Re: [Engine-devel] Proposal for new commit msg design for engine commits
This change to commit template has nothing to do with CI. it's a change that should reflect updated components relevance to the commit code.
This commit template is mostly invalid, as touching more than one 'subsystem' is possible, and has not enough granularity.
I suggest using a tag at the end with some extra syntax, like: Components: core, storage, db Components: all Components: all, !core, !db
For example, database change should trigger what?
All the jobs that are tagged for that component (db upgrades I suppose). And if the changes affect storage components then also storage, if the changes affect others then those others too.
Infra change should trigger what? The same, all the jobs that are tagged as infra.
A change of both user interface and command should trigger what? All the jobs tagged by user interface and/or command.
So you end up with:
userportal: storage: core: db: some message
As I suggested before, I think it's better if you end with a commit message like:
Some message
Components: userportal, storage, core, db
Actually it can be easily done with a tag in the gerrit comment instead of the commit message.
Just to make who happy?
The developer, the qe, the ci and the infra people. This mechanism is to avoid running all the tests all the time. Of course there are some times when all the tests should be run to make sure nothing else changed, but most times you just need to run part of them to make sure you did not break something obvious.
And maybe there are 50 tests of network, and you need only 5 of them for the specific change, how do you mark it, so now a developer need to know any test? what if you add one tomorrow which is relevant to a similar change? how do you inform the developer that now he needs 6?
As I said before, what the developer specifies is not a list of tests, but a list of components, that qe has to map to different sets of tests that can change with time. So specifying webadmin will run all the tests in that group, that might be only one, or 100, and might be increasing/decreasing with time transparently for the developer. Adding a new component is not common and there's no need to do it so frequently.
Why should it be the developer responsibility and not the quality ensuring engineer responsibility to determine which tests should run and when?
Of course it's the responsibility of the qe engineer to determine when and which tests should be run. But this is meant to be a new tool for the developer not a substitute for the full qe tests, so the developer can easily make sure that he's changes do not break anything obvious before starting the real tests (that will take more time and resources). The developer just adds some metadata so the qe engineer can decide which tests to run per patch, so it's on qe's hand in the end to decide if ignore or not the metadata and which tests to run.
As far as this template was not actually used for anything but humans, it was not that important, but once you formalize it as an interface, I step forward and state that the subject line is not the right tool for the task at hand (or any for this matter).
I agree with that, I think that it should be a tag similar to Change-Id, at the end of the commit message.
The fact that you have in each commit are the sources that are modified, all the other data is just plain noise. From the sources that are modified you should be able to derive a test plan with high chance that this test program will be correct. Human intervention should be possible by ordering special tests that are outside of the standard policy, for cases in which the standard policy of deriving tests from sources is too narrow.
That's just not true. The sources are complicated enough to make two changes in the same file to affect different components. Any reused code is prone to affect multiple components, making it really hard to determine by which changed files which tests to run. And if you go down to the function/class level it's even harder to decide and to maintain. And of course it's not human error free, as the metadata in the files/directories is defined and maintained by a human. And in my opinion is a lot harder to implement and maintain, and a lot less agile, and does not get rid of the human factor.
Once again...
1. Commit messages are not the place to specify metadata to interact with automation, it is a record for future reviewer.
Agree, that's why I suggested triggering it from a comment message. But that will require the developer another step after pushing.
2. The metadata within sources are the mean to automate the list of tests related to a specific source without human interaction on each commit.
The drawbacks are: - It needs a lot of maintenance
One time maintenance compared to per patch. Easier to enforce project policy, than relay on developer's policy.
It's one time big implementation and continued maintenance, each time a new file is added, or moved it has to be changed, or when a file is used in more than one place for the first time ...
- It require very modular code and
Right, we require this anyway.
I agree, but the ETA of that is >> ETA of any other option
- Locks the developer on which tests he want to run
As I wrote, it locks nothing, it is the baseline.
You can't avoid executing those tests, and if you want something else you have to do more than just push, that is not an advantage in front of any other solution.
3. If there is doubt from list of tests run them all, this is simple rule for automation.
Of course, but if there is too much doubt all that maintenance is useless.
So you suggest we run partial?
Yes, I though that is what this thread was for, to discuss the possibility of running partial tests on patches so we do not overload the infra while getting some feedback before running the whole set of tests.
based on whose decision? On the developers decision
can we trust him? Enough to run some tests on his patch proposal, yes. If we do not trust any developer we become closed to newcomers, and that's against any open source principle (and red hat's too) in my opinion.
4. The metadata within files are not the only way to order tests, one can do this manually via jenkins or any other mean as one can now.
As with any other solution.
Right, because of that I do not think this is emergency. Just have jenkins jobs and people can order tests based on gerrit urls...
The problem is that people can not control which tests get triggered, only which patch they run on. If you trigger from a gerrit url, all the tests that can be triggered will (for that repo, meaning engine/vdsm/scheduler...).
5. The metadata within files will help us to achieve other targets, such as automatic CC maintainers, verify that +2/-2 are set by authorized maintainer of component etc...
That's true, but I think that maybe putting metadata in each file is overkilling. One file in the root of the repo with a few lines should be enough for that afaik.
The metadata model I suggested was within each source and within directory. The collection is metadata of source + recursive meta data of directories.
At file put signature. At directory put a file, such as .ovirt.metadata with same fields.
I think that will lead to forgetting to update some sources as you are not sure where the metadata is, for example, a lot of people will forget to update file /a/b/c.txt and only update /a/.ovirt.metadata as it is not obvious where the metadata is held (I know, you can just execute a couple of command to find out, but in my experience people is lazy, and things that depend on people not being lazy get forgotten). Having only one place for the metadata helps prevent that though.
6. Gerrit 2.6 supports labels[1] which are actually what you are trying to achieve using the commit message because of lack of other solutions.
They are not fit for that, at least yet.
So maybe better if we can help gerrit to improve[1]?
Thanks! Maybe we can spend some effort to make that happen, anyone with java skills and willing to help reading this thread?
Until we have metadata within source, and we don't as we void discuss this for long time, and try to find manual workarounds and solutions.
Or until we have tags in the messages, for the same reasons. Don't forget that metadata within source is also a 'manual workaround or solution', as someone has to maintain all the metadata in all the files/directories (maybe for that purpose will be better to have just one file with all that metadata, depending on which level of granularity we need to map all the files).
I think best is to spread metadata within the entire tree, as then maintainer of file control the metadata, if file is copied/duplicated/moved/renamed it keeps the metadata.
I can see your point, but that requires a lot of maintenance as I see it...
Add label for each test, this will allow ordering tests via the gerrit web interface post submit.
Adding a comment will too.
So why relay on commit message, you can just ask people to comment:
OVIRT_ORDER_TEST: test1
Agree, the commit message tag was though to avoid having to go to gerrit page and create a new comment for each patch you submit. That step has to be done manually right now. Maybe we can add a git hook on the client side that detects the tag in the commit message, deletes it and posts it as a separated comment on gerrit?
After people start to do this over and over and over and over they will appreciate the need to add metadata into the source tree.
I think that having to maintain metadata in all the files is way more annoying, but yes, as you say adding metadata to each file gives a lot more information, but only if it's up to date.
Just like claiming that the source is out of date. If there is a mistake in metadata the maintainer will detect that when metadata is used.
I'm sorry, but I think that most of our code is out to date :s, meaning it's not what it should be (not modular enough, duplicated code, legacy code... I haven't heard of a developer sayiong, wow! this code is really good and well maintained, it's usually the other way around), and because of that I think that the metadata will have similar treatment, changing that is not an easy and fast task. Though I really believe that we will improve it. Maybe any other developers can supply their views on this?
In future, if this works out we can help gerrit to improve by enhancing the labels into free text/combo box etc... Or maybe try to do this now if you like to help out gerrit to improve...
That will make labels fit our purpose, and avoid having to add tags, metadata and all that, but it's not done yet. I'll gladly add that feature, but I'm not familiar with java or gerrit code, so I might not be the best person to do it, I'm sure that it will take a lot more time for me than the other options (Eyal, any comment?). If you are fluent with java and you want to help you are very welcome :)
You can help in making the metadata into the tree. This will be a great step forward.
Of course, if that's what gets decided I will happily help as much as needed.
Thanks, Alon
So to get this closed and start working in a solution I've created an etherpad: http://etherpad.ovirt.org/p/commit_message_design Please anyone with opinion add you comments and votes, maybe we can close this next week and start working! :) Should we add vdsm also to the thread? They also want a solution like this for their tests. ps. you can add another proposals if you have them, of course. -- David Caro Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
participants (11)
-
Alon Bar-Lev -
Antoni Segura Puimedon -
Dan Kenigsberg -
David Caro -
David Caro Estevez
-
Eyal Edri -
Fabian Deutsch -
Itamar Heim -
Mike Kolesnik -
Moran Goldboim -
Yair Zaslavsky