[Kimchi-devel] [PATCH] [Kimchi] Isolate unit tests execution.

[Aline Manera]

On 05/20/2016 09:10 AM, Paulo Ricardo Paz Vital wrote:
> To show how the new output will be if applied this patch, you can see one
> execution at http://paste.fedoraproject.org/368843/
>
> In the summary you can see that two temporary files are pointed and only these
> two files will be available after the execution (all other files were deleted).

So will I need to check file by file to get the error output?! Why not 
use a single file?

> On May 20 08:27AM, Paulo Ricardo Paz Vital wrote:
>> On May 18 06:24PM, Aline Manera wrote:
>>>
>>> On 05/16/2016 11:16 PM, pvital at linux.vnet.ibm.com wrote:
>>>> From: Paulo Vital <pvital at linux.vnet.ibm.com>
>>>>
>>>> This patch isolates each unit test execution reducing the occurrences of tests
>>>> using data from previous unit tests and making results not reliable.
>>>>
>>>> In addition, it also simplifies the output of tests execution, printing the tags
>>>> PASSED and FAILED and the error/failed banners only when necessary.
>>>>
>>>> Signed-off-by: Paulo Vital <pvital at linux.vnet.ibm.com>
>>>> ---
>>>>    tests/run_tests.sh.in | 53 +++++++++++++++++++++++++++++++++++++++------------
>>>>    1 file changed, 41 insertions(+), 12 deletions(-)
>>>>
>>>> diff --git a/tests/run_tests.sh.in b/tests/run_tests.sh.in
>>>> index 68dfa2e..d9c2661 100644
>>>> --- a/tests/run_tests.sh.in
>>>> +++ b/tests/run_tests.sh.in
>>>> @@ -21,6 +21,10 @@
>>>>    HAVE_UNITTEST=@HAVE_PYMOD_UNITTEST@
>>>>    PYTHON_VER=@PYTHON_VERSION@
>>>>
>>>> +RED='\033[0;31m'
>>>> +GREEN='\033[0;32m'
>>>> +NC='\033[0m' # No Color
>>>> +
>>>>    if [ "$1" = "-v" ]; then
>>>>        OPTS="-v"
>>>>        shift
>>>> @@ -40,19 +44,44 @@ else
>>>>        CMD="python -m unittest"
>>>>    fi
>>>>
>>>> -LIST=($ARGS)
>>>> -MODEL_LIST=()
>>>> -MOCK_LIST=()
>>>> -for ((i=0;i<${#LIST[@]};i++)); do
>>>> -
>>>> -    if [[ ${LIST[$i]} == test_model* ]]; then
>>>> -        MODEL_LIST+=(${LIST[$i]})
>>>> +NUM_TESTS=0
>>>> +TIME=0
>>>> +declare -A FAILED_UT    # dict to store failed unit tests and its OUTPUT_FILE
>>>> +for UT in $ARGS; do
>>>> +    OUTPUT_FILE=$(mktemp)
>>>> +    echo -n "***** Running unit test: $UT... "
>>>> +    # ../../../../../ refers to the project root
>>>> +    # ../../../../ refers to wok directory
>>>> +    # ../../../ refers to plugins directory
>>>> +    PYTHONPATH=../../../../../:../../../../:../../../ \
>>>> +    PYTHONWARNINGS="ignore:Unverified HTTPS request" \
>>>> +    $CMD $OPTS $UT &> $OUTPUT_FILE
>>> You can do:
>>>
>>> output=$($CMD $OPTS $UT)
>>>
>>> to get the test output. That way we don't need to create multiple temporary
>>> files.
>>>
>> Yeap, I can do this. However, the idea to have a temporary file is keep the full
>> output of the test case execution for future analysis, instead of check only the
>> console history.
>>
>> This patch, when an error or failure happens, prints only the 'banner' with the
>> test case error on user console. With the output stored in a file (temporary
>> because I delete the files which finishes with no errors) I can see it later to
>> fix the error.
>>
>>>> +    RES=$?
>>>> +    if [ $RES -ne 0 ]; then
>>>> +        # unit test has failed, so keep the OUTPUT_FILE and print the results
>>>> +        echo -e "\t ${RED}FAILED${NC}"
>>>> +        ERROR_LOG_BEGIN=$(grep -n "^\==*" $OUTPUT_FILE | head -n1 | cut -d":" -f1)
>>>> +        ERROR_LOG_END=$(cat $OUTPUT_FILE | wc -l)
>>>> +        tail -n $((${ERROR_LOG_END}-${ERROR_LOG_BEGIN}+1)) $OUTPUT_FILE
>>>> +        FAILED_UT+=([$UT]=$OUTPUT_FILE)
>>>> +        OUTPUT=$(grep "Ran" $OUTPUT_FILE)
>>> Following my suggestion to do not use temporary files, all the above block
>>> will be replace by:
>>>
>>> FAILED_UT+=$output
>>>
>> With this changes the full output of test execution will be printed in the
>> console. As I told above, this patch's idea is 'clean' the output printed in the
>> console by showing only the error/failure 'banner' and keeping the full
>> execution stack on a file.
>>
>>>>        else
>>>> -        MOCK_LIST+=(${LIST[$i]})
>>>> +        # unit test has passed, so print the results and delete the OUTPUT_FILE
>>>> +        OUTPUT=$(grep "Ran" $OUTPUT_FILE)
>>>> +        echo -e "\t ${GREEN}PASSED${NC} - $OUTPUT"
>>>> +        rm -rf $OUTPUT_FILE
>>>>        fi
>>>> +    TES=$(echo $OUTPUT | cut -d" " -f2)
>>>> +    NUM_TESTS=$(echo "$NUM_TESTS + $TES" | bc)
>>>> +    TES=$(echo $OUTPUT | cut -d" " -f5)
>>>> +    TIME=$(echo "$TIME + ${TES:0:-1}" | bc)
>>>>    done
>>>>
>>>> -# ../../../../../ refers to the project root
>>>> -# ../../../../ refers to wok directory
>>>> -# ../../../ refers to plugins directory
>>>> -PYTHONPATH=../../../../../:../../../../:../../../ PYTHONWARNINGS="ignore:Unverified HTTPS request" $CMD $OPTS ${MODEL_LIST[@]} ${MOCK_LIST[@]}
>>>> +# Print summary results
>>>> +echo -e "======================================================================"
>>>> +echo -e "===================== Kimchi Unit Tests Summary ======================"
>>>> +echo -e "Ran $NUM_TESTS tests in $TIME seconds."
>>>> +for i in "${!FAILED_UT[@]}"; do
>>>> +    FAIL_STATS=$(grep FAILED ${FAILED_UT[$i]} | cut -d" " -f2-4)
>>>> +    echo -e "$i FAILED: $FAIL_STATS - log available at ${FAILED_UT[$i]}"
>>>> +done
>>>> --
>>>> 2.5.5
>> --
>> Paulo Ricardo Paz Vital
>> Linux Technology Center, IBM Systems
>> http://www.ibm.com/linux/ltc/
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>
> --
> Paulo Ricardo Paz Vital
> Linux Technology Center, IBM Systems
> http://www.ibm.com/linux/ltc/