[Kimchi-devel] [PATCH] [Kimchi] Isolate unit tests execution.

[Aline Manera]

On 05/16/2016 11:16 PM, pvital at linux.vnet.ibm.com wrote:
> From: Paulo Vital <pvital at linux.vnet.ibm.com>
>
> This patch isolates each unit test execution reducing the occurrences of tests
> using data from previous unit tests and making results not reliable.
>
> In addition, it also simplifies the output of tests execution, printing the tags
> PASSED and FAILED and the error/failed banners only when necessary.
>
> Signed-off-by: Paulo Vital <pvital at linux.vnet.ibm.com>
> ---
>   tests/run_tests.sh.in | 53 +++++++++++++++++++++++++++++++++++++++------------
>   1 file changed, 41 insertions(+), 12 deletions(-)
>
> diff --git a/tests/run_tests.sh.in b/tests/run_tests.sh.in
> index 68dfa2e..d9c2661 100644
> --- a/tests/run_tests.sh.in
> +++ b/tests/run_tests.sh.in
> @@ -21,6 +21,10 @@
>   HAVE_UNITTEST=@HAVE_PYMOD_UNITTEST@
>   PYTHON_VER=@PYTHON_VERSION@
>
> +RED='\033[0;31m'
> +GREEN='\033[0;32m'
> +NC='\033[0m' # No Color
> +
>   if [ "$1" = "-v" ]; then
>       OPTS="-v"
>       shift
> @@ -40,19 +44,44 @@ else
>       CMD="python -m unittest"
>   fi
>
> -LIST=($ARGS)
> -MODEL_LIST=()
> -MOCK_LIST=()
> -for ((i=0;i<${#LIST[@]};i++)); do
> -
> -    if [[ ${LIST[$i]} == test_model* ]]; then
> -        MODEL_LIST+=(${LIST[$i]})
> +NUM_TESTS=0
> +TIME=0
> +declare -A FAILED_UT    # dict to store failed unit tests and its OUTPUT_FILE
> +for UT in $ARGS; do
> +    OUTPUT_FILE=$(mktemp)
> +    echo -n "***** Running unit test: $UT... "
> +    # ../../../../../ refers to the project root
> +    # ../../../../ refers to wok directory
> +    # ../../../ refers to plugins directory
> +    PYTHONPATH=../../../../../:../../../../:../../../ \
> +    PYTHONWARNINGS="ignore:Unverified HTTPS request" \
> +    $CMD $OPTS $UT &> $OUTPUT_FILE

You can do:

output=$($CMD $OPTS $UT)

to get the test output. That way we don't need to create multiple 
temporary files.

> +    RES=$?
> +    if [ $RES -ne 0 ]; then
> +        # unit test has failed, so keep the OUTPUT_FILE and print the results
> +        echo -e "\t ${RED}FAILED${NC}"

> +        ERROR_LOG_BEGIN=$(grep -n "^\==*" $OUTPUT_FILE | head -n1 | cut -d":" -f1)
> +        ERROR_LOG_END=$(cat $OUTPUT_FILE | wc -l)
> +        tail -n $((${ERROR_LOG_END}-${ERROR_LOG_BEGIN}+1)) $OUTPUT_FILE
> +        FAILED_UT+=([$UT]=$OUTPUT_FILE)
> +        OUTPUT=$(grep "Ran" $OUTPUT_FILE)

Following my suggestion to do not use temporary files, all the above 
block will be replace by:

FAILED_UT+=$output

>       else
> -        MOCK_LIST+=(${LIST[$i]})
> +        # unit test has passed, so print the results and delete the OUTPUT_FILE
> +        OUTPUT=$(grep "Ran" $OUTPUT_FILE)
> +        echo -e "\t ${GREEN}PASSED${NC} - $OUTPUT"
> +        rm -rf $OUTPUT_FILE
>       fi
> +    TES=$(echo $OUTPUT | cut -d" " -f2)
> +    NUM_TESTS=$(echo "$NUM_TESTS + $TES" | bc)
> +    TES=$(echo $OUTPUT | cut -d" " -f5)
> +    TIME=$(echo "$TIME + ${TES:0:-1}" | bc)
>   done
>
> -# ../../../../../ refers to the project root
> -# ../../../../ refers to wok directory
> -# ../../../ refers to plugins directory
> -PYTHONPATH=../../../../../:../../../../:../../../ PYTHONWARNINGS="ignore:Unverified HTTPS request" $CMD $OPTS ${MODEL_LIST[@]} ${MOCK_LIST[@]}
> +# Print summary results
> +echo -e "======================================================================"
> +echo -e "===================== Kimchi Unit Tests Summary ======================"
> +echo -e "Ran $NUM_TESTS tests in $TIME seconds."
> +for i in "${!FAILED_UT[@]}"; do
> +    FAIL_STATS=$(grep FAILED ${FAILED_UT[$i]} | cut -d" " -f2-4)
> +    echo -e "$i FAILED: $FAIL_STATS - log available at ${FAILED_UT[$i]}"
> +done
> --
> 2.5.5