test_cold_incremental_backup_vm2: "Cannot backup VM. The VM is during a backup operation"
On Sun, Jul 4, 2021 at 8:26 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 1291 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1291 [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2
Error Message: ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
Something similar now happened to me too: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/ https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/... This job triggered also he-basic, which passed. basic failed as in here. Any idea? Best regards,
Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208> get_vm_service_for_vm = <function get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268>
@order_by(_TEST_LIST) def test_cold_incremental_backup_vm2(engine_api, get_vm_service_for_vm): _verify_vm_state(engine_api.system_service(), VM2_NAME, types.VmStatus.DOWN) vm2_backups_service = get_vm_service_for_vm(VM2_NAME).backups_service() backup.perform_incremental_vm_backup(
engine_api, vm2_backups_service, DISK2_NAME, "cold_vm_backup")
basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/storage_utils/backup.py:75: in perform_incremental_vm_backup correlation_id="incremental_" + correlation_id) ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup ), query={'correlation_id': correlation_id} /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add return self._internal_add(backup, headers, query, wait) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in _internal_add return future.wait() if wait else future /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback self._check_fault(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in _check_fault self._raise_error(response, body) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
response = <ovirtsdk4.http.Response object at 0x7f1f94399c50> detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60>
@staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault.
This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None
msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message
if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.'
class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError
error = class_(msg) error.code = response.code if response else None error.fault = fault
raise error
E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWGVM5...
-- Didi
On Wed, Jul 7, 2021 at 11:18 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 4, 2021 at 8:26 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 1291 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1291 [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2
Error Message: ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
This is probably caused by changes in backup API in 4.4.7. The test is likely using the old API, which leads to timeout when waiting for the backup. In general, this test not implemented properly. It should use backup_vm.py script from the SDK, that was already updated to use the new API. Duplicating the backup code in OST means we need duplicate effort in maintaining the tests, and wasting tons on time on testing the changes in OST. The test should be skipped until someone can rewrite it properly.
Something similar now happened to me too:
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/...
This job triggered also he-basic, which passed. basic failed as in here.
Any idea?
Best regards,
Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208> get_vm_service_for_vm = <function get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268>
@order_by(_TEST_LIST) def test_cold_incremental_backup_vm2(engine_api, get_vm_service_for_vm): _verify_vm_state(engine_api.system_service(), VM2_NAME, types.VmStatus.DOWN) vm2_backups_service = get_vm_service_for_vm(VM2_NAME).backups_service() backup.perform_incremental_vm_backup(
engine_api, vm2_backups_service, DISK2_NAME, "cold_vm_backup")
basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/storage_utils/backup.py:75: in perform_incremental_vm_backup correlation_id="incremental_" + correlation_id) ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup ), query={'correlation_id': correlation_id} /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add return self._internal_add(backup, headers, query, wait) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in _internal_add return future.wait() if wait else future /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback self._check_fault(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in _check_fault self._raise_error(response, body) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
response = <ovirtsdk4.http.Response object at 0x7f1f94399c50> detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60>
@staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault.
This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None
msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message
if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.'
class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError
error = class_(msg) error.code = response.code if response else None error.fault = fault
raise error
E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWGVM5...
-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AVVGEGNJSNUJOG...
+Pavel Bar <pbar@redhat.com> On Wed, Jul 7, 2021 at 11:18 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 4, 2021 at 8:26 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 1291 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1291 [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2
Error Message: ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
Something similar now happened to me too:
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/...
This job triggered also he-basic, which passed. basic failed as in here.
Any idea?
Best regards,
Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208> get_vm_service_for_vm = <function
get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268>
@order_by(_TEST_LIST) def test_cold_incremental_backup_vm2(engine_api,
get_vm_service_for_vm):
_verify_vm_state(engine_api.system_service(), VM2_NAME,
types.VmStatus.DOWN)
vm2_backups_service =
get_vm_service_for_vm(VM2_NAME).backups_service()
backup.perform_incremental_vm_backup(
engine_api, vm2_backups_service, DISK2_NAME,
"cold_vm_backup")
basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ost_utils/ost_utils/storage_utils/backup.py:75: in
_ _ _ _ perform_incremental_vm_backup
correlation_id="incremental_" + correlation_id) ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup ), query={'correlation_id': correlation_id} /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add return self._internal_add(backup, headers, query, wait) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in
_internal_add
return future.wait() if wait else future /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback self._check_fault(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in
_check_fault
self._raise_error(response, body) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
response = <ovirtsdk4.http.Response object at 0x7f1f94399c50> detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60>
@staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given
HTTP
response and fault.
This method is intended for internal use by other components of
the
SDK. Refrain from using it directly, as backwards compatibility
isn't
guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None
msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' %
response.message
if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.'
class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError
error = class_(msg) error.code = response.code if response else None error.fault = fault
raise error
E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118:
Error_______________________________________________
Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWGVM5...
-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AVVGEGNJSNUJOG...
Hi, I think I discovered an issue that might explain the above failure. There is a flow with the new backup phases added by me, that when backup fails with some unexpected internal problem, the backup phase stays as some intermediate phase, i.e., "FINALIZING" instead of moving to "FAILED". I mean the backup command finishes (with failure), but the DB entry stays as non final one, neither "SUCCEEDED" nor "FAILED". Thus trying to execute another backup on the same VM fails immediately with the "Cannot backup VM. The VM is during a backup operation" error. That probably causes the above error. Also the full backup scenario executed via the SDK "backup_vm.py" script gets stuck on infinite loop waiting for either success or failure. That definitely needs to be fixed. I;m having a meeting with Benny soon. Pavel On Wed, 7 Jul 2021 at 23:33, Benny Zlotnik <bzlotnik@redhat.com> wrote:
+Pavel Bar <pbar@redhat.com>
On Wed, Jul 7, 2021 at 11:18 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 4, 2021 at 8:26 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 1291 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1291 [Marcin Sobczyk] network: Add missing 'ansible_hosts' fixture
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_004_basic_sanity.test_cold_incremental_backup_vm2
Error Message: ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
Something similar now happened to me too:
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/17729/...
This job triggered also he-basic, which passed. basic failed as in here.
Any idea?
Best regards,
Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f1f943c3208> get_vm_service_for_vm = <function
get_vm_service_for_vm.<locals>.service_for at 0x7f1f944e1268>
@order_by(_TEST_LIST) def test_cold_incremental_backup_vm2(engine_api,
get_vm_service_for_vm):
_verify_vm_state(engine_api.system_service(), VM2_NAME,
types.VmStatus.DOWN)
vm2_backups_service =
get_vm_service_for_vm(VM2_NAME).backups_service()
backup.perform_incremental_vm_backup(
engine_api, vm2_backups_service, DISK2_NAME,
"cold_vm_backup")
basic-suite-master/test-scenarios/test_004_basic_sanity.py:1056: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ost_utils/ost_utils/storage_utils/backup.py:75: in
_ _ _ _ perform_incremental_vm_backup
correlation_id="incremental_" + correlation_id) ost_utils/ost_utils/storage_utils/backup.py:34: in perform_vm_backup ), query={'correlation_id': correlation_id} /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:34139: in add return self._internal_add(backup, headers, query, wait) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: in
_internal_add
return future.wait() if wait else future /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: in callback self._check_fault(response) /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: in
_check_fault
self._raise_error(response, body) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
response = <ovirtsdk4.http.Response object at 0x7f1f94399c50> detail = <ovirtsdk4.types.Fault object at 0x7f1f94399f60>
@staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given
HTTP
response and fault.
This method is intended for internal use by other components of
the
SDK. Refrain from using it directly, as backwards compatibility
isn't
guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None
msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' %
response.message
if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.'
class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError
error = class_(msg) error.code = response.code if response else None error.fault = fault
raise error
E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409.
/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118:
Error_______________________________________________
Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCZSYD7BHWGVM5...
-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AVVGEGNJSNUJOG...
participants (4)
-
Benny Zlotnik -
Nir Soffer -
Pavel Bar -
Yedidyah Bar David