Re: [Kimchi-devel] [PATCH] [Kimchi] Isolate unit tests execution.
On May 18 06:24PM, Aline Manera wrote:
On 05/16/2016 11:16 PM, pvital@linux.vnet.ibm.com wrote:
From: Paulo Vital <pvital@linux.vnet.ibm.com>
This patch isolates each unit test execution reducing the occurrences of tests using data from previous unit tests and making results not reliable.
In addition, it also simplifies the output of tests execution, printing the tags PASSED and FAILED and the error/failed banners only when necessary.
Signed-off-by: Paulo Vital <pvital@linux.vnet.ibm.com> --- tests/run_tests.sh.in | 53 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 12 deletions(-)
diff --git a/tests/run_tests.sh.in b/tests/run_tests.sh.in index 68dfa2e..d9c2661 100644 --- a/tests/run_tests.sh.in +++ b/tests/run_tests.sh.in @@ -21,6 +21,10 @@ HAVE_UNITTEST=@HAVE_PYMOD_UNITTEST@ PYTHON_VER=@PYTHON_VERSION@
+RED='\033[0;31m' +GREEN='\033[0;32m' +NC='\033[0m' # No Color + if [ "$1" = "-v" ]; then OPTS="-v" shift @@ -40,19 +44,44 @@ else CMD="python -m unittest" fi
-LIST=($ARGS) -MODEL_LIST=() -MOCK_LIST=() -for ((i=0;i<${#LIST[@]};i++)); do - - if [[ ${LIST[$i]} == test_model* ]]; then - MODEL_LIST+=(${LIST[$i]}) +NUM_TESTS=0 +TIME=0 +declare -A FAILED_UT # dict to store failed unit tests and its OUTPUT_FILE +for UT in $ARGS; do + OUTPUT_FILE=$(mktemp) + echo -n "***** Running unit test: $UT... " + # ../../../../../ refers to the project root + # ../../../../ refers to wok directory + # ../../../ refers to plugins directory + PYTHONPATH=../../../../../:../../../../:../../../ \ + PYTHONWARNINGS="ignore:Unverified HTTPS request" \ + $CMD $OPTS $UT &> $OUTPUT_FILE
You can do:
output=$($CMD $OPTS $UT)
to get the test output. That way we don't need to create multiple temporary files.
Yeap, I can do this. However, the idea to have a temporary file is keep the full output of the test case execution for future analysis, instead of check only the console history. This patch, when an error or failure happens, prints only the 'banner' with the test case error on user console. With the output stored in a file (temporary because I delete the files which finishes with no errors) I can see it later to fix the error.
+ RES=$? + if [ $RES -ne 0 ]; then + # unit test has failed, so keep the OUTPUT_FILE and print the results + echo -e "\t ${RED}FAILED${NC}"
+ ERROR_LOG_BEGIN=$(grep -n "^\==*" $OUTPUT_FILE | head -n1 | cut -d":" -f1) + ERROR_LOG_END=$(cat $OUTPUT_FILE | wc -l) + tail -n $((${ERROR_LOG_END}-${ERROR_LOG_BEGIN}+1)) $OUTPUT_FILE + FAILED_UT+=([$UT]=$OUTPUT_FILE) + OUTPUT=$(grep "Ran" $OUTPUT_FILE)
Following my suggestion to do not use temporary files, all the above block will be replace by:
FAILED_UT+=$output
With this changes the full output of test execution will be printed in the console. As I told above, this patch's idea is 'clean' the output printed in the console by showing only the error/failure 'banner' and keeping the full execution stack on a file.
else - MOCK_LIST+=(${LIST[$i]}) + # unit test has passed, so print the results and delete the OUTPUT_FILE + OUTPUT=$(grep "Ran" $OUTPUT_FILE) + echo -e "\t ${GREEN}PASSED${NC} - $OUTPUT" + rm -rf $OUTPUT_FILE fi + TES=$(echo $OUTPUT | cut -d" " -f2) + NUM_TESTS=$(echo "$NUM_TESTS + $TES" | bc) + TES=$(echo $OUTPUT | cut -d" " -f5) + TIME=$(echo "$TIME + ${TES:0:-1}" | bc) done
-# ../../../../../ refers to the project root -# ../../../../ refers to wok directory -# ../../../ refers to plugins directory -PYTHONPATH=../../../../../:../../../../:../../../ PYTHONWARNINGS="ignore:Unverified HTTPS request" $CMD $OPTS ${MODEL_LIST[@]} ${MOCK_LIST[@]} +# Print summary results +echo -e "======================================================================" +echo -e "===================== Kimchi Unit Tests Summary ======================" +echo -e "Ran $NUM_TESTS tests in $TIME seconds." +for i in "${!FAILED_UT[@]}"; do + FAIL_STATS=$(grep FAILED ${FAILED_UT[$i]} | cut -d" " -f2-4) + echo -e "$i FAILED: $FAIL_STATS - log available at ${FAILED_UT[$i]}" +done -- 2.5.5
-- Paulo Ricardo Paz Vital Linux Technology Center, IBM Systems http://www.ibm.com/linux/ltc/
To show how the new output will be if applied this patch, you can see one execution at http://paste.fedoraproject.org/368843/ In the summary you can see that two temporary files are pointed and only these two files will be available after the execution (all other files were deleted). On May 20 08:27AM, Paulo Ricardo Paz Vital wrote:
On May 18 06:24PM, Aline Manera wrote:
On 05/16/2016 11:16 PM, pvital@linux.vnet.ibm.com wrote:
From: Paulo Vital <pvital@linux.vnet.ibm.com>
This patch isolates each unit test execution reducing the occurrences of tests using data from previous unit tests and making results not reliable.
In addition, it also simplifies the output of tests execution, printing the tags PASSED and FAILED and the error/failed banners only when necessary.
Signed-off-by: Paulo Vital <pvital@linux.vnet.ibm.com> --- tests/run_tests.sh.in | 53 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 12 deletions(-)
diff --git a/tests/run_tests.sh.in b/tests/run_tests.sh.in index 68dfa2e..d9c2661 100644 --- a/tests/run_tests.sh.in +++ b/tests/run_tests.sh.in @@ -21,6 +21,10 @@ HAVE_UNITTEST=@HAVE_PYMOD_UNITTEST@ PYTHON_VER=@PYTHON_VERSION@
+RED='\033[0;31m' +GREEN='\033[0;32m' +NC='\033[0m' # No Color + if [ "$1" = "-v" ]; then OPTS="-v" shift @@ -40,19 +44,44 @@ else CMD="python -m unittest" fi
-LIST=($ARGS) -MODEL_LIST=() -MOCK_LIST=() -for ((i=0;i<${#LIST[@]};i++)); do - - if [[ ${LIST[$i]} == test_model* ]]; then - MODEL_LIST+=(${LIST[$i]}) +NUM_TESTS=0 +TIME=0 +declare -A FAILED_UT # dict to store failed unit tests and its OUTPUT_FILE +for UT in $ARGS; do + OUTPUT_FILE=$(mktemp) + echo -n "***** Running unit test: $UT... " + # ../../../../../ refers to the project root + # ../../../../ refers to wok directory + # ../../../ refers to plugins directory + PYTHONPATH=../../../../../:../../../../:../../../ \ + PYTHONWARNINGS="ignore:Unverified HTTPS request" \ + $CMD $OPTS $UT &> $OUTPUT_FILE
You can do:
output=$($CMD $OPTS $UT)
to get the test output. That way we don't need to create multiple temporary files.
Yeap, I can do this. However, the idea to have a temporary file is keep the full output of the test case execution for future analysis, instead of check only the console history.
This patch, when an error or failure happens, prints only the 'banner' with the test case error on user console. With the output stored in a file (temporary because I delete the files which finishes with no errors) I can see it later to fix the error.
+ RES=$? + if [ $RES -ne 0 ]; then + # unit test has failed, so keep the OUTPUT_FILE and print the results + echo -e "\t ${RED}FAILED${NC}"
+ ERROR_LOG_BEGIN=$(grep -n "^\==*" $OUTPUT_FILE | head -n1 | cut -d":" -f1) + ERROR_LOG_END=$(cat $OUTPUT_FILE | wc -l) + tail -n $((${ERROR_LOG_END}-${ERROR_LOG_BEGIN}+1)) $OUTPUT_FILE + FAILED_UT+=([$UT]=$OUTPUT_FILE) + OUTPUT=$(grep "Ran" $OUTPUT_FILE)
Following my suggestion to do not use temporary files, all the above block will be replace by:
FAILED_UT+=$output
With this changes the full output of test execution will be printed in the console. As I told above, this patch's idea is 'clean' the output printed in the console by showing only the error/failure 'banner' and keeping the full execution stack on a file.
else - MOCK_LIST+=(${LIST[$i]}) + # unit test has passed, so print the results and delete the OUTPUT_FILE + OUTPUT=$(grep "Ran" $OUTPUT_FILE) + echo -e "\t ${GREEN}PASSED${NC} - $OUTPUT" + rm -rf $OUTPUT_FILE fi + TES=$(echo $OUTPUT | cut -d" " -f2) + NUM_TESTS=$(echo "$NUM_TESTS + $TES" | bc) + TES=$(echo $OUTPUT | cut -d" " -f5) + TIME=$(echo "$TIME + ${TES:0:-1}" | bc) done
-# ../../../../../ refers to the project root -# ../../../../ refers to wok directory -# ../../../ refers to plugins directory -PYTHONPATH=../../../../../:../../../../:../../../ PYTHONWARNINGS="ignore:Unverified HTTPS request" $CMD $OPTS ${MODEL_LIST[@]} ${MOCK_LIST[@]} +# Print summary results +echo -e "======================================================================" +echo -e "===================== Kimchi Unit Tests Summary ======================" +echo -e "Ran $NUM_TESTS tests in $TIME seconds." +for i in "${!FAILED_UT[@]}"; do + FAIL_STATS=$(grep FAILED ${FAILED_UT[$i]} | cut -d" " -f2-4) + echo -e "$i FAILED: $FAIL_STATS - log available at ${FAILED_UT[$i]}" +done -- 2.5.5
-- Paulo Ricardo Paz Vital Linux Technology Center, IBM Systems http://www.ibm.com/linux/ltc/
_______________________________________________ Kimchi-devel mailing list Kimchi-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/kimchi-devel
-- Paulo Ricardo Paz Vital Linux Technology Center, IBM Systems http://www.ibm.com/linux/ltc/
participants (1)
-
Paulo Ricardo Paz Vital