- Infra - oVirt List Archives

[JIRA] (OVIRT-2193) vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory
by Dafna Ron (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2193?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2193: ---------------------------------- [~dfediuck] [~eedri], [~gbenhaim(a)redhat.com] and I were discussing this yesterday. The majority of the current failed tests that are marked as "race" are caused due to locks and tasks cleanup in engine and we should not be failing the tests if they are modified to query tasks/locked objects before command is run and fail on query timeout. The Network team fixed their testes which were failing on similar issues and I am happy to say it reduced these sort of failures for network. The majority of these failures today are in storage related tests which [~tnisan(a)redhat.com] has been notified and has assigned [~derez(a)redhat.com] to look at. However, if I continue to see these tests failing sporadically (and I mean a test that is failing sporadically over a period of a month) I may have to skip those tests until they are fixed. We can improve on deciding what is an expectable % of failures and for what period of time we should quantify it before we skip the tests and decide on a cooperation with developers regarding the tests ownership (i.e maybe have a point of contact on each team for ost) > vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory > ---------------------------------------------------------------------- > > Key: OVIRT-2193 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2193 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we have a failed vdsm test in basic master suite. > 004_basic_sanity.preview_snapshot_with_memory > seems like a regression. > ' 2018-06-12 07:31:08,080-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-27) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Input validation failed while processing 'POST' request for path '/vms/ec6dcfb7-dda9-4c58-b048-87b08ffcbfbe > /previewsnapshot'. > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Exception: org.ovirt.api.metamodel.server.ValidationException: Parameter 'snapshot.id' is mandatory but was > not provided. > at org.ovirt.engine.api.resource.VmResourceHelper.validatePreviewSnapshot(VmResourceHelper.java:75) [restapi-definition.jar:] > at org.ovirt.engine.api.resource.VmResource.doPreviewSnapshot(VmResource.java:333) [restapi-definition.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_171] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_171] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_171] > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]' > Change 90532,7 (vdsm) is probably the reason behind recent system test failures > in the "ovirt-master" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/90532/7 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8169/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 323 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Jul '18

11 Jul '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 323 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #323 [Sandro Bonazzola] master: consume qemu-kvm-ev test builds ----------------- Failed Tests: ----------------- All tests passed

1 1

[JIRA] (OVIRT-2193) vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2193?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2193: ---------------------------------- We need to try to formalize some process around it, I'm not talking about one-time failures, its tests that fail every once in a while. There are not "environment" issues, this is isolated Lago run, so the chance of that happening is very low. We also filter out any issues that are infra related, so only if a real code race happens, we open a ticket and mark it as 'race'. If developers need more info/logs/insights then they should ask for it, otherwise, I think that we'll just start disabling tests that fail too often, this is taking a heavy toll on monitoring and manual work from the team, we need to improve the current handling of race failures. > vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory > ---------------------------------------------------------------------- > > Key: OVIRT-2193 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2193 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we have a failed vdsm test in basic master suite. > 004_basic_sanity.preview_snapshot_with_memory > seems like a regression. > ' 2018-06-12 07:31:08,080-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-27) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Input validation failed while processing 'POST' request for path '/vms/ec6dcfb7-dda9-4c58-b048-87b08ffcbfbe > /previewsnapshot'. > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Exception: org.ovirt.api.metamodel.server.ValidationException: Parameter 'snapshot.id' is mandatory but was > not provided. > at org.ovirt.engine.api.resource.VmResourceHelper.validatePreviewSnapshot(VmResourceHelper.java:75) [restapi-definition.jar:] > at org.ovirt.engine.api.resource.VmResource.doPreviewSnapshot(VmResource.java:333) [restapi-definition.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_171] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_171] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_171] > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]' > Change 90532,7 (vdsm) is probably the reason behind recent system test failures > in the "ovirt-master" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/90532/7 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8169/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2193) vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory
by Doron Fediuck (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2193?page=com.atlassian.jira.… ] Doron Fediuck commented on OVIRT-2193: -------------------------------------- A race by definition may or may not materialize. If it's consistent that's an easy thing to fix. If it's not, We need some more insights than the end result. For example, is there an environmental issue, etc. > vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory > ---------------------------------------------------------------------- > > Key: OVIRT-2193 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2193 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we have a failed vdsm test in basic master suite. > 004_basic_sanity.preview_snapshot_with_memory > seems like a regression. > ' 2018-06-12 07:31:08,080-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-27) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Input validation failed while processing 'POST' request for path '/vms/ec6dcfb7-dda9-4c58-b048-87b08ffcbfbe > /previewsnapshot'. > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Exception: org.ovirt.api.metamodel.server.ValidationException: Parameter 'snapshot.id' is mandatory but was > not provided. > at org.ovirt.engine.api.resource.VmResourceHelper.validatePreviewSnapshot(VmResourceHelper.java:75) [restapi-definition.jar:] > at org.ovirt.engine.api.resource.VmResource.doPreviewSnapshot(VmResource.java:333) [restapi-definition.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_171] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_171] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_171] > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]' > Change 90532,7 (vdsm) is probably the reason behind recent system test failures > in the "ovirt-master" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/90532/7 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8169/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2055) failure in suspend resume vm not related to reported change - need to investigate
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2055?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2055: ---------------------------------- Did you ask someone from virt team? was [~mskrivanek] notified on this? > failure in suspend resume vm not related to reported change - need to investigate > --------------------------------------------------------------------------------- > > Key: OVIRT-2055 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2055 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > Change 91507,1 (ovirt-engine-sdk-java) is probably the reason behind recent > system test failures in the "ovirt-master" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/91507/1 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/7721/ > I think the issue is on qemu/libvirt changed queries: > https://pastebin.com/afyeUAcT > https://pastebin.com/va20Y5Hp > checking his with vdsm people. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2193) vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2193?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2193: ---------------------------------- [~dfediuck] we seem to be getting race failures like that often, which makes it very hard to enable reporting back to Gerrit to finding a resolution. We can't continue with the approach of "no reason for it to fail, its a race" and stop the debugging process there. There has to be a more in depth investigation from development side if we want to start reducing these races. > vdsm project CQ failure: 004_basic_sanity.preview_snapshot_with_memory > ---------------------------------------------------------------------- > > Key: OVIRT-2193 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2193 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we have a failed vdsm test in basic master suite. > 004_basic_sanity.preview_snapshot_with_memory > seems like a regression. > ' 2018-06-12 07:31:08,080-04 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-27) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Input validation failed while processing 'POST' request for path '/vms/ec6dcfb7-dda9-4c58-b048-87b08ffcbfbe > /previewsnapshot'. > 2018-06-12 07:31:08,087-04 ERROR [org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper] (default task-27) [] Exception: org.ovirt.api.metamodel.server.ValidationException: Parameter 'snapshot.id' is mandatory but was > not provided. > at org.ovirt.engine.api.resource.VmResourceHelper.validatePreviewSnapshot(VmResourceHelper.java:75) [restapi-definition.jar:] > at org.ovirt.engine.api.resource.VmResource.doPreviewSnapshot(VmResource.java:333) [restapi-definition.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_171] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_171] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_171] > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]' > Change 90532,7 (vdsm) is probably the reason behind recent system test failures > in the "ovirt-master" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/90532/7 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8169/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2195) race in 004_basic_sanity.verify_vm_import on master CQ
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2195?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2195: ---------------------------------- Another open isssue that is open for almost a month, we have to see how we shorten the cycle here. Dafna, please escalate if you don't get a reply or resolution from the relevant maintainer after a few days. If Shmuel is checking it, he needs to assigned to this ticket and provide a result after a few days max. FYI [~dfediuck] > race in 004_basic_sanity.verify_vm_import on master CQ > ------------------------------------------------------ > > Key: OVIRT-2195 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2195 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > We have a race that causes test: 004_basic_sanity.verify_vm_import to fail > This build: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8183/ failed but there is nothing in the log to suggest that the import failed. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2202) 006_migrations.prepare_migration_attachments_ipv4 for ovirt-engine-dashboard
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2202?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2202: ---------------------------------- This is open for almost a month, I don't think we still even have logs to look at them? [~dron] if you can't get a resolution or response on an error, please let me know on the same week, so we can escalate. Right now you can probably close it as can't reproduce. > 006_migrations.prepare_migration_attachments_ipv4 for ovirt-engine-dashboard > ---------------------------------------------------------------------------- > > Key: OVIRT-2202 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2202 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > There was a failed test 006_migrations.prepare_migration_attachments_ipv4 > Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML > Build: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8208/ > patch reported: https://gerrit.ovirt.org/#/c/92238/1 - add resolutions section to force jquery and bootstrap versions > The actual error is: Unable to process messages Broken pipe: java.io.IOException: Broken pipe > this would be the error in vdsm: > '2018-06-14 04:05:34,698-0400 ERROR (jsonrpc/5) [vds] connectivity check failed (API:1537) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1534, in setupNetworks > supervdsm.getProxy().setupNetworks(networks, bondings, options) > File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ > return callMethod() > File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> > **kwargs) > File "<string>", line 2, in setupNetworks > File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod > raise convert_to_error(kind, result) > ConfigNetworkError: (10, 'connectivity check failed') > 2018-06-14 04:05:34,699-0400 INFO (jsonrpc/5) [api.network] FINISH setupNetworks return={'status': {'message': 'connectivity check failed', 'code': 10}} from=::ffff:192.168.201.4,50952 (api:53) > 2018-06-14 04:05:34,699-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error 10) in 127.20 seconds (__init__:311)' > The patch may be related so I will as Greg to take a look -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2203) failure in update host caused patch to fail
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2203?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2203: ---------------------------------- [~sbonazzo(a)redhat.com] who can look at the upgrade logs and see if there is a bug there? > failure in update host caused patch to fail > ------------------------------------------- > > Key: OVIRT-2203 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2203 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra, ost_race > > The patch failed due to failure to update the host: > '2018-06-14 08:25:38,754-04 ERROR [org.ovirt.engine.core.bll.host.HostUpgradeManager] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [cb22fd71-b94d-4567-a8e4-78238868d2de] Failed to run check-update of host 'lago-basic-suite-master- > host-0'' > I can see that the upgrade suite also failed for not seeing repos > The next metrics build succeeded successfully so there is no need to do anything > I cannot be sure but I think its probably a network glitch or jenkins issue which happened during the test run. > here is the info from the mail: > Change 92195,2 (ovirt-engine-metrics) is probably the reason behind recent > system test failures in the "ovirt-master" change queue and needs to be fixed. > This change had been removed from the testing queue. Artifacts build from this > change will not be released until it is fixed. > For further details about the change see: > https://gerrit.ovirt.org/#/c/92195/2 > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8213/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0

[JIRA] (OVIRT-2264) Cannot deactivate Storage while there are running tasks on this Storage
by Eyal Edri (oVirt JIRA) 11 Jul '18

11 Jul '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2264?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2264: ---------------------------------- [~derez@redhat.com][~tnisan@redhat.com] any update? > Cannot deactivate Storage while there are running tasks on this Storage > ----------------------------------------------------------------------- > > Key: OVIRT-2264 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2264 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_race > > we failed test: 007_sd_reattach.deactivate_storage_domain in vdsm project with error: Cannot deactivate Storage while there are running tasks on this Storage. -Please wait until tasks will finish and try again.]". HTTP response code is 409. > I am opening this as a race in case something changed in OST tests which would cause this to repeat. > https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2506 > https://gerrit.ovirt.org/#/c/92313/2 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100088)

1 0