Hi,
I think I discovered an issue that might explain the above failure.
There is a flow with the new backup phases added by me, that when backup
fails with some unexpected internal problem, the backup phase stays as some
intermediate phase, i.e., "FINALIZING" instead of moving to "FAILED".
I mean the backup command finishes (with failure), but the DB entry stays
as non final one, neither "SUCCEEDED" nor "FAILED".
Thus trying to execute another backup on the same VM fails immediately with
the "Cannot backup VM. The VM is during a backup operation" error.
That probably causes the above error.
Also the full backup scenario executed via the SDK "backup_vm.py" script
gets stuck on infinite loop waiting for either success or failure.
That definitely needs to be fixed.
I;m having a meeting with Benny soon.
Pavel
On Wed, 7 Jul 2021 at 23:33, Benny Zlotnik <bzlotnik(a)redhat.com> wrote:
+Pavel Bar <pbar(a)redhat.com>
On Wed, Jul 7, 2021 at 11:18 PM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> On Sun, Jul 4, 2021 at 8:26 AM <jenkins(a)jenkins.phx.ovirt.org> wrote:
> >
> > Project:
>
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
> > Build:
>
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night...
> > Build Number: 1291
> > Build Status: Failure
> > Triggered By: Started by timer
> >
> > -------------------------------------
> > Changes Since Last Success:
> > -------------------------------------
> > Changes for Build #1291
> > [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture
> >
> >
> >
> >
> > -----------------
> > Failed Tests:
> > -----------------
> > 1 tests failed.
> > FAILED:
>
basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2
> >
> > Error Message:
> > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
> "[Cannot backup VM. The VM is during a backup operation.]". HTTP response
> code is 409.
>
> Something similar now happened to me too:
>
>
>
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/
>
>
>
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/177...
>
> This job triggered also he-basic, which passed. basic failed as in here.
>
> Any idea?
>
> Best regards,
>
> >
> > Stack Trace:
> > engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208>
> > get_vm_service_for_vm = <function
> get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268>
> >
> > @order_by(_TEST_LIST)
> > def test_cold_incremental_backup_vm2(engine_api,
> get_vm_service_for_vm):
> > _verify_vm_state(engine_api.system_service(), VM2_NAME,
> types.VmStatus.DOWN)
> > vm2_backups_service =
> get_vm_service_for_vm(VM2_NAME).backups_service()
> > backup.perform_incremental_vm_backup(
> > > engine_api, vm2_backups_service, DISK2_NAME,
> "cold_vm_backup")
> >
> > basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056:
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _
> > ost_utils/ost_utils/storage_utils/backup.py:75: in
> perform_incremental_vm_backup
> > correlation_id="incremental_" + correlation_id)
> > ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup
> > ), query={'correlation_id': correlation_id}
> > /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add
> > return self._internal_add(backup, headers, query, wait)
> > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in
> _internal_add
> > return future.wait() if wait else future
> > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait
> > return self._code(response)
> > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback
> > self._check_fault(response)
> > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in
> _check_fault
> > self._raise_error(response, body)
> > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _
> >
> > response = <ovirtsdk4.http.Response object at 0x7f1f94399c50>
> > detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60>
> >
> > @staticmethod
> > def _raise_error(response, detail=None):
> > """
> > Creates and raises an error containing the details of the given
> HTTP
> > response and fault.
> >
> > This method is intended for internal use by other components of
> the
> > SDK. Refrain from using it directly, as backwards compatibility
> isn't
> > guaranteed.
> > """
> > fault = detail if isinstance(detail, types.Fault) else None
> >
> > msg = ''
> > if fault:
> > if fault.reason:
> > if msg:
> > msg += ' '
> > msg = msg + 'Fault reason is "%s".' %
fault.reason
> > if fault.detail:
> > if msg:
> > msg += ' '
> > msg = msg + 'Fault detail is "%s".' %
fault.detail
> > if response:
> > if response.code:
> > if msg:
> > msg += ' '
> > msg = msg + 'HTTP response code is %s.' % response.code
> > if response.message:
> > if msg:
> > msg += ' '
> > msg = msg + 'HTTP response message is "%s".'
%
> response.message
> >
> > if isinstance(detail, six.string_types):
> > if msg:
> > msg += ' '
> > msg = msg + detail + '.'
> >
> > class_ = Error
> > if response is not None:
> > if response.code in [401, 403]:
> > class_ = AuthError
> > elif response.code == 404:
> > class_ = NotFoundError
> >
> > error = class_(msg)
> > error.code = response.code if response else None
> > error.fault = fault
> > > raise error
> > E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
> detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP
> response code is 409.
> >
> > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118:
> Error_______________________________________________
> > Infra mailing list -- infra(a)ovirt.org
> > To unsubscribe send an email to infra-leave(a)ovirt.org
> > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
>
https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
>
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWG...
>
>
>
> --
> Didi
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org
> To unsubscribe send an email to devel-leave(a)ovirt.org
> Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
>
https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AVVGEGNJSNU...
>