Hi,
I think I discovered an issue that might explain the above failure.
There is a flow with the new backup phases added by me, that when backup fails with some unexpected internal problem, the backup phase stays as some intermediate phase, i.e., "FINALIZING" instead of moving to "FAILED".
I mean the backup command finishes (with failure), but the DB entry stays as non final one, neither "SUCCEEDED" nor "FAILED".
Thus trying to execute another backup on the same VM fails immediately with the "Cannot backup VM. The VM is during a backup operation" error.
That probably causes the above error.
Also the full backup scenario executed via the SDK "backup_vm.py" script gets stuck on infinite loop waiting for either success or failure.
That definitely needs to be fixed.

I;m having a meeting with Benny soon.

Pavel


On Wed, 7 Jul 2021 at 23:33, Benny Zlotnik <bzlotnik@redhat.com> wrote:
+Pavel Bar 

On Wed, Jul 7, 2021 at 11:18 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 4, 2021 at 8:26 AM <jenkins@jenkins.phx.ovirt.org> wrote:
>
> Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
> Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/1291/
> Build Number: 1291
> Build Status:  Failure
> Triggered By: Started by timer
>
> -------------------------------------
> Changes Since Last Success:
> -------------------------------------
> Changes for Build #1291
> [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture
>
>
>
>
> -----------------
> Failed Tests:
> -----------------
> 1 tests failed.
> FAILED:  basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2
>
> Error Message:
> ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.

Something similar now happened to me too:

https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/

https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/testReport/junit/basic-suite-master.test-scenarios/test_004_basic_sanity/Invoking_jobs___check_patch_basic_suite_master_el8_x86_64___test_cold_incremental_backup_vm2/

This job triggered also he-basic, which passed. basic failed as in here.

Any idea?

Best regards,

>
> Stack Trace:
> engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208>
> get_vm_service_for_vm = <function get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268>
>
>     @order_by(_TEST_LIST)
>     def test_cold_incremental_backup_vm2(engine_api, get_vm_service_for_vm):
>         _verify_vm_state(engine_api.system_service(), VM2_NAME, types.VmStatus.DOWN)
>         vm2_backups_service = get_vm_service_for_vm(VM2_NAME).backups_service()
>         backup.perform_incremental_vm_backup(
> >           engine_api, vm2_backups_service, DISK2_NAME, "cold_vm_backup")
>
> basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ost_utils/ost_utils/storage_utils/backup.py:75: in perform_incremental_vm_backup
>     correlation_id="incremental_" + correlation_id)
> ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup
>     ), query={'correlation_id': correlation_id}
> /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add
>     return self._internal_add(backup, headers, query, wait)
> /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in _internal_add
>     return future.wait() if wait else future
> /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait
>     return self._code(response)
> /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback
>     self._check_fault(response)
> /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in _check_fault
>     self._raise_error(response, body)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> response = <ovirtsdk4.http.Response object at 0x7f1f94399c50>
> detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60>
>
>     @staticmethod
>     def _raise_error(response, detail=None):
>         """
>         Creates and raises an error containing the details of the given HTTP
>         response and fault.
>
>         This method is intended for internal use by other components of the
>         SDK. Refrain from using it directly, as backwards compatibility isn't
>         guaranteed.
>         """
>         fault = detail if isinstance(detail, types.Fault) else None
>
>         msg = ''
>         if fault:
>             if fault.reason:
>                 if msg:
>                     msg += ' '
>                 msg = msg + 'Fault reason is "%s".' % fault.reason
>             if fault.detail:
>                 if msg:
>                     msg += ' '
>                 msg = msg + 'Fault detail is "%s".' % fault.detail
>         if response:
>             if response.code:
>                 if msg:
>                     msg += ' '
>                 msg = msg + 'HTTP response code is %s.' % response.code
>             if response.message:
>                 if msg:
>                     msg += ' '
>                 msg = msg + 'HTTP response message is "%s".' % response.message
>
>         if isinstance(detail, six.string_types):
>             if msg:
>                 msg += ' '
>             msg = msg + detail + '.'
>
>         class_ = Error
>         if response is not None:
>             if response.code in [401, 403]:
>                 class_ = AuthError
>             elif response.code == 404:
>                 class_ = NotFoundError
>
>         error = class_(msg)
>         error.code = response.code if response else None
>         error.fault = fault
> >       raise error
> E       ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
>
> /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error_______________________________________________
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWGVM5RZN7LWPJVMOKKGRK54/



--
Didi
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AVVGEGNJSNUJOGXPBXD2KUL7I7PY7XKC/