[Kimchi-devel] [kimchi-devel] FVT Regression test suite for Kimchi 1.5

Tue Aug 11 14:54:19 UTC 2015

On 10/08/2015 03:46, Chandra Shehkhar Reddy Potula wrote:
>
>
> On 08/06/2015 11:51 PM, Aline Manera wrote:
>>
>>
>> On 06/08/2015 04:06, Chandra Shehkhar Reddy Potula wrote:
>>>
>>> Hi Aline,
>>>
>>> Thanks for your inputs !!!
>>>
>>> The tests we written so far are functional verification test cases 
>>> which can be ran against the Kimchi which is installed, up and 
>>> running and can validate REST API calls with the real data.
>>
>> What do you mean by a "Kimchi installed, up and running"? Is it not 
>> assuming a local instance running from local source code?
> I mean, in order to run FVT test suite, Kimchi must be installed on 
> one of the supported distro.  Test suite may not be on the same target 
> machine where it is suppose to run but it can also be triggered from 
> different machine against the target machine.
>>
>>>
>>>  About the current test cases under folder*/tests*, If I understood 
>>> correctly they are suppose to do unit testing, hence majority of 
>>> them uses mocking. I went through test cases like 
>>> test_authorization.py, test_exception.py, test_host.py, 
>>> test_plugin.py, test_server.py etc (by the way I excluded mockmodel 
>>> and model test files) and I saw the *import kimchi.mockmodel, *which 
>>> makes me feel that these cases are designed for mock testing. They 
>>> are not meant for functional verification even though they are REST 
>>> API calls.
>>>
>>
>> The MockModel is to allow testing functionalities in which we may 
>> depend on system configuration.
>>
>> For example, how test authorization if you don't know which users and 
>> groups exists on the system? Even though you know the username, what 
>> about the password? How will you test login?
>> A similar scenario applies to other tests we use MockModel.
>>
>> Also MockModel also connects to libvirt to do the virtualization 
>> work. The difference is it uses the libvirt Test driver instead of 
>> the qemu one.
>> And as the libvirt Test driver has some limitations we needed to 
>> overwrite some functions. It is better to understand looking at the 
>> code in mockmodel.py.
>>
>> Could you give me an example of a FVT test you have in mind? Just to 
>> confirm we don't already have some of them upstream.
> Exactly, Mock is an object that simulates the behavior of a real 
> object in controlled ways. Mock objects are used in unit testing.
> With MockModel it still helps you in testing certain functionality but 
> on simulated data/environment.
>
> On the other hand Functional Verification Test - (also can be called 
> as Integration Test), helps testing a slice of functionality on a real 
> system. This will test many methods and may interact with dependencies 
> involved.
>
> We maintained a config file, which has a section to provide the 
> session details (user, host, ssl port num etc) of the target machine 
> against which test suite will run. End user have to update the config 
> file before executing the test suite.
>
> For example FVT test cases we developed:
> Test case to perform Virtual Machine life-cycle operations.
> Test case for debug report generations.
> Test case for repository creation/updation/deletion.
> Test case for create/modify/update various template scenario's.
> Test case to create/activate/de-activate/delete of network interfaces 
> like isolation, NAT and bridge network interfaces.
> Test case to create/activate/deactivate/delete of storage pools of 
> various types (DIR, SCSI, iSCSI, Logical) and storage volumes 
> operations (create/update/delete) on each of these storage pools.
> and so on....

Well, we've already had those tests in our current master branch. 
Please, check all test_model* files under /tests directory. All those 
tests are made on real Model environment.

>>
>>> I agree that we could place functional verification test suite under 
>>> the*/tests* folder, but not to mix with existing unit test cases. I 
>>> do recommend below structure inside */tests.*
>>>
>>> /tests
>>> *|**
>>> **-----*/tests-unit
>>> *|**
>>> **-----*/tests-fvt
>>>
>>> /ui/tests
>>> *|**
>>> **-----*/tests-unit
>>> *|**
>>> **-----*/tests -selinium
>>
>> I'd only suggest to remove the second "tests-" as it seems redundant 
>> for me =)
>>
>> /tests/unit/
>> /tests/fvt
>>
>> /ui/tests/unit
>> /ui/tests/selenium
>>
> Fine with me !!!
>>>
>>> Hope this make sense !!!
>>>
>>>
>>> Regards
>>> Chandra
>>> On 08/06/2015 12:35 AM, Aline Manera wrote:
>>>>
>>>>
>>>> On 04/08/2015 07:22, Chandra Sr Potula wrote:
>>>>>
>>>>> We have developed functional verification regression test suite 
>>>>> for Kimchi/Ginger in order to verify the REST API's calls 
>>>>> currently available on 1.5.
>>>>>
>>>>> Our proposal is to create the following folder structure in the 
>>>>> main stream of Kimchi/Ginger for FVT. From this week we will be 
>>>>> submitting the patches.
>>>>>
>>>>> *tests-fvt*
>>>>>    |
>>>>>    -----*tests-restapi*
>>>>>    |
>>>>>    -----*tests-ui*
>>>>>
>>>>> Please provide feedback if you have any !!!
>>>>>
>>>>
>>>> We already have unit tests to cover the REST APIs. You can check 
>>>> them under /tests
>>>> You can add as many tests as you want there. Usually we are on a 
>>>> way to remove test_model and test_mockmodel and separate the tests 
>>>> per components: templates, networks, guests, etc.
>>>> About the directory structure, I don't think we need to change it.
>>>>
>>>> For UI tests, we need to wait a little bit to do that according to 
>>>> the new UI.
>>>> Samuel will do unit tests and Selenium tests as soon as the new UI 
>>>> components are getting ready.
>>>> About the directory structure, we can create a new directory under 
>>>> /ui, for example, ui/tests to handle all the unit tests and 
>>>> ui/tests/selenium to handle the Selenium tests.
>>>>
>>>>>
>>>>> Thanks in advance.
>>>>> Chandra
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Kimchi-devel mailing list
>>>>> Kimchi-devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Kimchi-devel mailing list
>>>> Kimchi-devel at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/kimchi-devel/attachments/20150811/4dc16a83/attachment.html>