[Kimchi-devel] [PATCH] [Kimchi] Isolate unit tests execution.

[Aline Manera]

On 05/20/2016 08:27 AM, Paulo Ricardo Paz Vital wrote:
> On May 18 06:24PM, Aline Manera wrote:
>>
>> On 05/16/2016 11:16 PM, pvital at linux.vnet.ibm.com wrote:
>>> From: Paulo Vital <pvital at linux.vnet.ibm.com>
>>>
>>> This patch isolates each unit test execution reducing the occurrences of tests
>>> using data from previous unit tests and making results not reliable.
>>>
>>> In addition, it also simplifies the output of tests execution, printing the tags
>>> PASSED and FAILED and the error/failed banners only when necessary.
>>>
>>> Signed-off-by: Paulo Vital <pvital at linux.vnet.ibm.com>
>>> ---
>>>    tests/run_tests.sh.in | 53 +++++++++++++++++++++++++++++++++++++++------------
>>>    1 file changed, 41 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/tests/run_tests.sh.in b/tests/run_tests.sh.in
>>> index 68dfa2e..d9c2661 100644
>>> --- a/tests/run_tests.sh.in
>>> +++ b/tests/run_tests.sh.in
>>> @@ -21,6 +21,10 @@
>>>    HAVE_UNITTEST=@HAVE_PYMOD_UNITTEST@
>>>    PYTHON_VER=@PYTHON_VERSION@
>>>
>>> +RED='\033[0;31m'
>>> +GREEN='\033[0;32m'
>>> +NC='\033[0m' # No Color
>>> +
>>>    if [ "$1" = "-v" ]; then
>>>        OPTS="-v"
>>>        shift
>>> @@ -40,19 +44,44 @@ else
>>>        CMD="python -m unittest"
>>>    fi
>>>
>>> -LIST=($ARGS)
>>> -MODEL_LIST=()
>>> -MOCK_LIST=()
>>> -for ((i=0;i<${#LIST[@]};i++)); do
>>> -
>>> -    if [[ ${LIST[$i]} == test_model* ]]; then
>>> -        MODEL_LIST+=(${LIST[$i]})
>>> +NUM_TESTS=0
>>> +TIME=0
>>> +declare -A FAILED_UT    # dict to store failed unit tests and its OUTPUT_FILE
>>> +for UT in $ARGS; do
>>> +    OUTPUT_FILE=$(mktemp)
>>> +    echo -n "***** Running unit test: $UT... "
>>> +    # ../../../../../ refers to the project root
>>> +    # ../../../../ refers to wok directory
>>> +    # ../../../ refers to plugins directory
>>> +    PYTHONPATH=../../../../../:../../../../:../../../ \
>>> +    PYTHONWARNINGS="ignore:Unverified HTTPS request" \
>>> +    $CMD $OPTS $UT &> $OUTPUT_FILE
>> You can do:
>>
>> output=$($CMD $OPTS $UT)
>>
>> to get the test output. That way we don't need to create multiple temporary
>> files.
>>
> Yeap, I can do this. However, the idea to have a temporary file is keep the full
> output of the test case execution for future analysis, instead of check only the
> console history.
>
> This patch, when an error or failure happens, prints only the 'banner' with the
> test case error on user console. With the output stored in a file (temporary
> because I delete the files which finishes with no errors) I can see it later to
> fix the error.
>

My point is: you don't need to have multiple files.
Store the error output in a variable and if it is not empty in the end 
of the process, write it into a file and print the output to user as you 
already do.

>>> +    RES=$?
>>> +    if [ $RES -ne 0 ]; then
>>> +        # unit test has failed, so keep the OUTPUT_FILE and print the results
>>> +        echo -e "\t ${RED}FAILED${NC}"
>>> +        ERROR_LOG_BEGIN=$(grep -n "^\==*" $OUTPUT_FILE | head -n1 | cut -d":" -f1)
>>> +        ERROR_LOG_END=$(cat $OUTPUT_FILE | wc -l)
>>> +        tail -n $((${ERROR_LOG_END}-${ERROR_LOG_BEGIN}+1)) $OUTPUT_FILE
>>> +        FAILED_UT+=([$UT]=$OUTPUT_FILE)
>>> +        OUTPUT=$(grep "Ran" $OUTPUT_FILE)
>> Following my suggestion to do not use temporary files, all the above block
>> will be replace by:
>>
>> FAILED_UT+=$output
>>
> With this changes the full output of test execution will be printed in the
> console. As I told above, this patch's idea is 'clean' the output printed in the
> console by showing only the error/failure 'banner' and keeping the full
> execution stack on a file.
>
>>>        else
>>> -        MOCK_LIST+=(${LIST[$i]})
>>> +        # unit test has passed, so print the results and delete the OUTPUT_FILE
>>> +        OUTPUT=$(grep "Ran" $OUTPUT_FILE)
>>> +        echo -e "\t ${GREEN}PASSED${NC} - $OUTPUT"
>>> +        rm -rf $OUTPUT_FILE
>>>        fi
>>> +    TES=$(echo $OUTPUT | cut -d" " -f2)
>>> +    NUM_TESTS=$(echo "$NUM_TESTS + $TES" | bc)
>>> +    TES=$(echo $OUTPUT | cut -d" " -f5)
>>> +    TIME=$(echo "$TIME + ${TES:0:-1}" | bc)
>>>    done
>>>
>>> -# ../../../../../ refers to the project root
>>> -# ../../../../ refers to wok directory
>>> -# ../../../ refers to plugins directory
>>> -PYTHONPATH=../../../../../:../../../../:../../../ PYTHONWARNINGS="ignore:Unverified HTTPS request" $CMD $OPTS ${MODEL_LIST[@]} ${MOCK_LIST[@]}
>>> +# Print summary results
>>> +echo -e "======================================================================"
>>> +echo -e "===================== Kimchi Unit Tests Summary ======================"
>>> +echo -e "Ran $NUM_TESTS tests in $TIME seconds."
>>> +for i in "${!FAILED_UT[@]}"; do
>>> +    FAIL_STATS=$(grep FAILED ${FAILED_UT[$i]} | cut -d" " -f2-4)
>>> +    echo -e "$i FAILED: $FAIL_STATS - log available at ${FAILED_UT[$i]}"
>>> +done
>>> --
>>> 2.5.5
> --
> Paulo Ricardo Paz Vital
> Linux Technology Center, IBM Systems
> http://www.ibm.com/linux/ltc/