4:22 a.m.
We have developed functional verification regression test suite for
Kimchi/Ginger in order to verify the REST API's calls currently available
on 1.5.
Our proposal is to create the following folder structure in the main stream
of Kimchi/Ginger for FVT. From this week we will be submitting the patches.
tests-fvt
|
-----tests-restapi
|
-----tests-ui
Please provide feedback if you have any !!!
Thanks in advance.
Chandra
1:05 p.m.
On 04/08/2015 07:22, Chandra Sr Potula wrote:
We have developed functional verification regression test suite for
Kimchi/Ginger in order to verify the REST API's calls currently
available on 1.5.
Our proposal is to create the following folder structure in the main
stream of Kimchi/Ginger for FVT. From this week we will be submitting
the patches.
*tests-fvt*
|
-----*tests-restapi*
|
-----*tests-ui*
Please provide feedback if you have any !!!
We already have unit tests to cover the REST APIs. You can check them
under /tests
You can add as many tests as you want there. Usually we are on a way to
remove test_model and test_mockmodel and separate the tests per
components: templates, networks, guests, etc.
About the directory structure, I don't think we need to change it.
For UI tests, we need to wait a little bit to do that according to the
new UI.
Samuel will do unit tests and Selenium tests as soon as the new UI
components are getting ready.
About the directory structure, we can create a new directory under /ui,
for example, ui/tests to handle all the unit tests and ui/tests/selenium
to handle the Selenium tests.
Thanks in advance.
Chandra
_______________________________________________
Kimchi-devel mailing list
Kimchi-devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel
1:06 a.m.
Hi Aline,
Thanks for your inputs !!!
The tests we written so far are functional verification test cases which
can be ran against the Kimchi which is installed, up and running and can
validate REST API calls with the real data.
About the current test cases under folder*/tests*, If I understood
correctly they are suppose to do unit testing, hence majority of them
uses mocking. I went through test cases like test_authorization.py,
test_exception.py, test_host.py, test_plugin.py, test_server.py etc (by
the way I excluded mockmodel and model test files) and I saw the *import
kimchi.mockmodel, *which makes me feel that these cases are designed for
mock testing. They are not meant for functional verification even though
they are REST API calls.
I agree that we could place functional verification test suite under
the*/tests* folder, but not to mix with existing unit test cases. I do
recommend below structure inside */tests.*
/tests
*|**
**-----*/tests-unit
*|**
**-----*/tests-fvt
/ui/tests
*|**
**-----*/tests-unit
*|**
**-----*/tests -selinium
Hope this make sense !!!
Regards
Chandra
On 08/06/2015 12:35 AM, Aline Manera wrote:
On 04/08/2015 07:22, Chandra Sr Potula wrote:
>
> We have developed functional verification regression test suite for
> Kimchi/Ginger in order to verify the REST API's calls currently
> available on 1.5.
>
> Our proposal is to create the following folder structure in the main
> stream of Kimchi/Ginger for FVT. From this week we will be submitting
> the patches.
>
> *tests-fvt*
> |
> -----*tests-restapi*
> |
> -----*tests-ui*
>
> Please provide feedback if you have any !!!
>
We already have unit tests to cover the REST APIs. You can check them
under /tests
You can add as many tests as you want there. Usually we are on a way
to remove test_model and test_mockmodel and separate the tests per
components: templates, networks, guests, etc.
About the directory structure, I don't think we need to change it.
For UI tests, we need to wait a little bit to do that according to the
new UI.
Samuel will do unit tests and Selenium tests as soon as the new UI
components are getting ready.
About the directory structure, we can create a new directory under
/ui, for example, ui/tests to handle all the unit tests and
ui/tests/selenium to handle the Selenium tests.
>
> Thanks in advance.
> Chandra
>
>
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
_______________________________________________
Kimchi-devel mailing list
Kimchi-devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel
12:21 p.m.
On 06/08/2015 04:06, Chandra Shehkhar Reddy Potula wrote:
Hi Aline,
Thanks for your inputs !!!
The tests we written so far are functional verification test cases
which can be ran against the Kimchi which is installed, up and running
and can validate REST API calls with the real data.
What do you mean by a "Kimchi installed, up and running"? Is it not
assuming a local instance running from local source code?
About the current test cases under folder*/tests*, If I understood
correctly they are suppose to do unit testing, hence majority of them
uses mocking. I went through test cases like test_authorization.py,
test_exception.py, test_host.py, test_plugin.py, test_server.py etc
(by the way I excluded mockmodel and model test files) and I saw the
*import kimchi.mockmodel, *which makes me feel that these cases are
designed for mock testing. They are not meant for functional
verification even though they are REST API calls.
The MockModel is to allow testing functionalities in which we may depend
on system configuration.
For example, how test authorization if you don't know which users and
groups exists on the system? Even though you know the username, what
about the password? How will you test login?
A similar scenario applies to other tests we use MockModel.
Also MockModel also connects to libvirt to do the virtualization work.
The difference is it uses the libvirt Test driver instead of the qemu one.
And as the libvirt Test driver has some limitations we needed to
overwrite some functions. It is better to understand looking at the code
in mockmodel.py.
Could you give me an example of a FVT test you have in mind? Just to
confirm we don't already have some of them upstream.
I agree that we could place functional verification test suite under
the*/tests* folder, but not to mix with existing unit test cases. I do
recommend below structure inside */tests.*
/tests
*|**
**-----*/tests-unit
*|**
**-----*/tests-fvt
/ui/tests
*|**
**-----*/tests-unit
*|**
**-----*/tests -selinium
I'd only suggest to remove the second "tests-" as it seems redundant for
me =)
/tests/unit/
/tests/fvt
/ui/tests/unit
/ui/tests/selenium
Hope this make sense !!!
Regards
Chandra
On 08/06/2015 12:35 AM, Aline Manera wrote:
>
>
> On 04/08/2015 07:22, Chandra Sr Potula wrote:
>>
>> We have developed functional verification regression test suite for
>> Kimchi/Ginger in order to verify the REST API's calls currently
>> available on 1.5.
>>
>> Our proposal is to create the following folder structure in the main
>> stream of Kimchi/Ginger for FVT. From this week we will be
>> submitting the patches.
>>
>> *tests-fvt*
>> |
>> -----*tests-restapi*
>> |
>> -----*tests-ui*
>>
>> Please provide feedback if you have any !!!
>>
>
> We already have unit tests to cover the REST APIs. You can check them
> under /tests
> You can add as many tests as you want there. Usually we are on a way
> to remove test_model and test_mockmodel and separate the tests per
> components: templates, networks, guests, etc.
> About the directory structure, I don't think we need to change it.
>
> For UI tests, we need to wait a little bit to do that according to
> the new UI.
> Samuel will do unit tests and Selenium tests as soon as the new UI
> components are getting ready.
> About the directory structure, we can create a new directory under
> /ui, for example, ui/tests to handle all the unit tests and
> ui/tests/selenium to handle the Selenium tests.
>
>>
>> Thanks in advance.
>> Chandra
>>
>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>
>
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
12:46 a.m.
On 08/06/2015 11:51 PM, Aline Manera wrote:
On 06/08/2015 04:06, Chandra Shehkhar Reddy Potula wrote:
>
> Hi Aline,
>
> Thanks for your inputs !!!
>
> The tests we written so far are functional verification test cases
> which can be ran against the Kimchi which is installed, up and
> running and can validate REST API calls with the real data.
What do you mean by a "Kimchi installed, up and running"? Is it not
assuming a local instance running from local source code?
I mean, in order to run
FVT test suite, Kimchi must be installed on one
of the supported distro. Test suite may not be on the same target
machine where it is suppose to run but it can also be triggered from
different machine against the target machine.
>
> About the current test cases under folder*/tests*, If I understood
> correctly they are suppose to do unit testing, hence majority of them
> uses mocking. I went through test cases like test_authorization.py,
> test_exception.py, test_host.py, test_plugin.py, test_server.py etc
> (by the way I excluded mockmodel and model test files) and I saw the
> *import kimchi.mockmodel, *which makes me feel that these cases are
> designed for mock testing. They are not meant for functional
> verification even though they are REST API calls.
>
The MockModel is to allow testing functionalities in which we may
depend on system configuration.
For example, how test authorization if you don't know which users and
groups exists on the system? Even though you know the username, what
about the password? How will you test login?
A similar scenario applies to other tests we use MockModel.
Also MockModel also connects to libvirt to do the virtualization work.
The difference is it uses the libvirt Test driver instead of the qemu one.
And as the libvirt Test driver has some limitations we needed to
overwrite some functions. It is better to understand looking at the
code in mockmodel.py.
Could you give me an example of a FVT test you have in mind? Just to
confirm we don't already have some of them upstream.
Exactly, Mock is an object
that simulates the behavior of a real object
in controlled ways. Mock objects are used in unit testing.
With MockModel it still helps you in testing certain functionality but
on simulated data/environment.
On the other hand Functional Verification Test - (also can be called as
Integration Test), helps testing a slice of functionality on a real
system. This will test many methods and may interact with dependencies
involved.
We maintained a config file, which has a section to provide the session
details (user, host, ssl port num etc) of the target machine against
which test suite will run. End user have to update the config file
before executing the test suite.
For example FVT test cases we developed:
Test case to perform Virtual Machine life-cycle operations.
Test case for debug report generations.
Test case for repository creation/updation/deletion.
Test case for create/modify/update various template scenario's.
Test case to create/activate/de-activate/delete of network interfaces
like isolation, NAT and bridge network interfaces.
Test case to create/activate/deactivate/delete of storage pools of
various types (DIR, SCSI, iSCSI, Logical) and storage volumes operations
(create/update/delete) on each of these storage pools.
and so on....
> I agree that we could place functional verification test suite under
> the*/tests* folder, but not to mix with existing unit test cases. I
> do recommend below structure inside */tests.*
>
> /tests
> *|**
> **-----*/tests-unit
> *|**
> **-----*/tests-fvt
>
> /ui/tests
> *|**
> **-----*/tests-unit
> *|**
> **-----*/tests -selinium
I'd only suggest to remove the second "tests-" as it seems redundant
for me =)
/tests/unit/
/tests/fvt
/ui/tests/unit
/ui/tests/selenium
Fine with me !!!
>
> Hope this make sense !!!
>
>
> Regards
> Chandra
> On 08/06/2015 12:35 AM, Aline Manera wrote:
>>
>>
>> On 04/08/2015 07:22, Chandra Sr Potula wrote:
>>>
>>> We have developed functional verification regression test suite for
>>> Kimchi/Ginger in order to verify the REST API's calls currently
>>> available on 1.5.
>>>
>>> Our proposal is to create the following folder structure in the
>>> main stream of Kimchi/Ginger for FVT. From this week we will be
>>> submitting the patches.
>>>
>>> *tests-fvt*
>>> |
>>> -----*tests-restapi*
>>> |
>>> -----*tests-ui*
>>>
>>> Please provide feedback if you have any !!!
>>>
>>
>> We already have unit tests to cover the REST APIs. You can check
>> them under /tests
>> You can add as many tests as you want there. Usually we are on a way
>> to remove test_model and test_mockmodel and separate the tests per
>> components: templates, networks, guests, etc.
>> About the directory structure, I don't think we need to change it.
>>
>> For UI tests, we need to wait a little bit to do that according to
>> the new UI.
>> Samuel will do unit tests and Selenium tests as soon as the new UI
>> components are getting ready.
>> About the directory structure, we can create a new directory under
>> /ui, for example, ui/tests to handle all the unit tests and
>> ui/tests/selenium to handle the Selenium tests.
>>
>>>
>>> Thanks in advance.
>>> Chandra
>>>
>>>
>>>
>>> _______________________________________________
>>> Kimchi-devel mailing list
>>> Kimchi-devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>
>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>
8:54 a.m.
On 10/08/2015 03:46, Chandra Shehkhar Reddy Potula wrote:
On 08/06/2015 11:51 PM, Aline Manera wrote:
>
>
> On 06/08/2015 04:06, Chandra Shehkhar Reddy Potula wrote:
>>
>> Hi Aline,
>>
>> Thanks for your inputs !!!
>>
>> The tests we written so far are functional verification test cases
>> which can be ran against the Kimchi which is installed, up and
>> running and can validate REST API calls with the real data.
>
> What do you mean by a "Kimchi installed, up and running"? Is it not
> assuming a local instance running from local source code?
I mean, in order to run FVT test suite, Kimchi must be installed on
one of the supported distro. Test suite may not be on the same target
machine where it is suppose to run but it can also be triggered from
different machine against the target machine.
>
>>
>> About the current test cases under folder*/tests*, If I understood
>> correctly they are suppose to do unit testing, hence majority of
>> them uses mocking. I went through test cases like
>> test_authorization.py, test_exception.py, test_host.py,
>> test_plugin.py, test_server.py etc (by the way I excluded mockmodel
>> and model test files) and I saw the *import kimchi.mockmodel, *which
>> makes me feel that these cases are designed for mock testing. They
>> are not meant for functional verification even though they are REST
>> API calls.
>>
>
> The MockModel is to allow testing functionalities in which we may
> depend on system configuration.
>
> For example, how test authorization if you don't know which users and
> groups exists on the system? Even though you know the username, what
> about the password? How will you test login?
> A similar scenario applies to other tests we use MockModel.
>
> Also MockModel also connects to libvirt to do the virtualization
> work. The difference is it uses the libvirt Test driver instead of
> the qemu one.
> And as the libvirt Test driver has some limitations we needed to
> overwrite some functions. It is better to understand looking at the
> code in mockmodel.py.
>
> Could you give me an example of a FVT test you have in mind? Just to
> confirm we don't already have some of them upstream.
Exactly, Mock is an object that simulates the behavior of a real
object in controlled ways. Mock objects are used in unit testing.
With MockModel it still helps you in testing certain functionality but
on simulated data/environment.
On the other hand Functional Verification Test - (also can be called
as Integration Test), helps testing a slice of functionality on a real
system. This will test many methods and may interact with dependencies
involved.
We maintained a config file, which has a section to provide the
session details (user, host, ssl port num etc) of the target machine
against which test suite will run. End user have to update the config
file before executing the test suite.
For example FVT test cases we developed:
Test case to perform Virtual Machine life-cycle operations.
Test case for debug report generations.
Test case for repository creation/updation/deletion.
Test case for create/modify/update various template scenario's.
Test case to create/activate/de-activate/delete of network interfaces
like isolation, NAT and bridge network interfaces.
Test case to create/activate/deactivate/delete of storage pools of
various types (DIR, SCSI, iSCSI, Logical) and storage volumes
operations (create/update/delete) on each of these storage pools.
and so on....
Well, we've already had those tests in our current master branch.
Please, check all test_model* files under /tests directory. All those
tests are made on real Model environment.
>
>> I agree that we could place functional verification test suite under
>> the*/tests* folder, but not to mix with existing unit test cases. I
>> do recommend below structure inside */tests.*
>>
>> /tests
>> *|**
>> **-----*/tests-unit
>> *|**
>> **-----*/tests-fvt
>>
>> /ui/tests
>> *|**
>> **-----*/tests-unit
>> *|**
>> **-----*/tests -selinium
>
> I'd only suggest to remove the second "tests-" as it seems redundant
> for me =)
>
> /tests/unit/
> /tests/fvt
>
> /ui/tests/unit
> /ui/tests/selenium
>
Fine with me !!!
>>
>> Hope this make sense !!!
>>
>>
>> Regards
>> Chandra
>> On 08/06/2015 12:35 AM, Aline Manera wrote:
>>>
>>>
>>> On 04/08/2015 07:22, Chandra Sr Potula wrote:
>>>>
>>>> We have developed functional verification regression test suite
>>>> for Kimchi/Ginger in order to verify the REST API's calls
>>>> currently available on 1.5.
>>>>
>>>> Our proposal is to create the following folder structure in the
>>>> main stream of Kimchi/Ginger for FVT. From this week we will be
>>>> submitting the patches.
>>>>
>>>> *tests-fvt*
>>>> |
>>>> -----*tests-restapi*
>>>> |
>>>> -----*tests-ui*
>>>>
>>>> Please provide feedback if you have any !!!
>>>>
>>>
>>> We already have unit tests to cover the REST APIs. You can check
>>> them under /tests
>>> You can add as many tests as you want there. Usually we are on a
>>> way to remove test_model and test_mockmodel and separate the tests
>>> per components: templates, networks, guests, etc.
>>> About the directory structure, I don't think we need to change it.
>>>
>>> For UI tests, we need to wait a little bit to do that according to
>>> the new UI.
>>> Samuel will do unit tests and Selenium tests as soon as the new UI
>>> components are getting ready.
>>> About the directory structure, we can create a new directory under
>>> /ui, for example, ui/tests to handle all the unit tests and
>>> ui/tests/selenium to handle the Selenium tests.
>>>
>>>>
>>>> Thanks in advance.
>>>> Chandra
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Kimchi-devel mailing list
>>>> Kimchi-devel(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>>
>>>
>>>
>>> _______________________________________________
>>> Kimchi-devel mailing list
>>> Kimchi-devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>
>
3178
days inactive
3185
days old
5 comments
3 participants
participants (3)
-
Aline Manera -
Chandra Shehkhar Reddy Potula -
Chandra Sr Potula