- Devel - oVirt List Archives

[ OST Failure Report ] [ oVirt Master (ovirt-engine-nodejs-modules) ] [ 27-02-2019 ] [ 002_bootstrap.add_vm2_lease ]
by Dafna Ron 10 Mar '19

10 Mar '19

Hi, We have a failure on the project in basic suite, master branch. The recent failure was in patch: https://gerrit.ovirt.org/#/c/98087/1 - Add pre-seed for ovirt-web-ui CQ is pointing at the below as the root cause (which was merged a while back): https://gerrit.ovirt.org/#/c/97491/ - Add pre-seed for ovirt-web-ui Can you please check the issue as it seems both patches are changing the same thing and the project seem to be broken since https://gerrit.ovirt.org/#/c/97491/3 Latest failure: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/ Logs: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13171/artifa… errors from logs: Engine: 2019-02-27 13:37:28,479-05 ERROR [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [74283e25] Transaction rolled-back for command 'org.ovirt.engine.core.bll.UpdateVmCommand'. 2019-02-27 13:37:28,483-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [74283e25] EVENT_ID: USER_FAILED_UPDATE_VM(58), Failed to update VM vm2 (User: admin@inter nal-authz). 2019-02-27 13:37:28,485-05 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [74283e25] Lock freed to object 'EngineLock:{exclusiveLocks='[vm2=VM_NAME]', sharedLocks='[3500eb82-e5e2-4e24-b41c-ea 02d9f6adee=VM]'}' 2019-02-27 13:37:28,485-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [74283e25] method: runAction, params: [UpdateVm, VmManagementParametersBase:{commandId='340 59769-05b9-429e-8356-f6b9b9953f55', user='admin', commandType='UpdateVm', vmId='3500eb82-e5e2-4e24-b41c-ea02d9f6adee'}], timeElapsed: 6618ms 2019-02-27 13:37:28,486-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [] 2019-02-27 13:37:28,579-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] method: get, params: [e29c0ba1-464c-4eb4-a8f2-c6933d9 9969a], timeElapsed: 3ms vdsm: 2019-02-27 13:37:21,987-0500 INFO (jsonrpc/1) [vdsm.api] FINISH lease_info error=No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee from=::ffff:192.168.201.4,43920, flow_id=117dec74-ad59-4b12-8148-b2c130337c10, task_id=9c297d41-0aa7-4c74-b268-b710e666bc6c (api:52) 2019-02-27 13:37:21,988-0500 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='9c297d41-0aa7-4c74-b268-b710e666bc6c') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in lease_info File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3702, in lease_info info = dom.lease_info(lease.lease_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 674, in lease_info return vol.lookup(lease_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 553, in lookup raise NoSuchLease(lease_id) NoSuchLease: No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee 2019-02-27 13:37:21,988-0500 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='9c297d41-0aa7-4c74-b268-b710e666bc6c') aborting: Task is aborted: u'No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee' - code 100 (task:1181) 2019-02-27 13:37:21,988-0500 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH lease_info error=No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error NoSuchLease: No such lease 3500eb82-e5e2-4e24-b41c-ea02d9f6adee

5 9

ovirt-vmconsole-1.0.6-3.fc28.noarch: Non-fatal POSTUN scriptlet failure
by Nir Soffer 10 Mar '19

10 Mar '19

When updating Fedora 28 host with ovirt-release-master repo, I see this error: Running scriptlet: ovirt-vmconsole-1.0.6-3.fc28.noarch 143/251 libsemanage.semanage_module_key_set_name: Name /usr/share/selinux/packages/ovirt-vmconsole/ovirt_vmconsole.pp is invalid. semodule: Failed on /usr/share/selinux/packages/ovirt-vmconsole/ovirt_vmconsole.pp! warning: %postun(ovirt-vmconsole-1.0.6-3.fc28.noarch) scriptlet failed, exit status 1 Non-fatal POSTUN scriptlet failure in rpm package ovirt-vmconsole Non-fatal POSTUN scriptlet failure in rpm package ovirt-vmconsole I did not try to use it, I don't know if we have a real issue. Looks like we need to either fix the issue or silence the warnings. Nir

2 1

[Ovirt] [CQ weekly status] [08-03-2019]
by Dafna Ron 09 Mar '19

09 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 06-03-2019 on project ovirt-ansible-vm-infra due to missing jtk2 package. this issue is now fixed with patch: https://gerrit.ovirt.org/#/c/98304/ *CQ-4.3*: RED (#1) Last job to fail is for project imgbased. the failure is due to the random failures we have been having from test add_vm2_lease. this issue is currently debated in the list. New repo and packages were updated by Galit due to failure in CQ for missing packages: https://gerrit.ovirt.org/#/c/98312/ *CQ-Master:* RED (#1) We were failing this week due to missing packages which was fixed by: https://gerrit.ovirt.org/#/c/98304/ We also had two regressions which were fixed by: https://gerrit.ovirt.org/#/c/98317/ and by https://gerrit.ovirt.org/#/c/98371/ All projects are currently passing Current running jobs for 4.2 [1] and master [2] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 1

ovirt and AMD virtualization bug
by Hetz Ben Hamo 09 Mar '19

09 Mar '19

Hi, I've done some research on a bug which I found on the AMD Zen/Zen+ based CPU's. Apparently, when using nested virtualization, the CPU doesn't expose the "monitor" CPU flag. While other virtualization solutions (Xen, ESXi, Hyper-V) don't care about this issue and let you run with nested without any problem, oVirt doesn't let you launch or create any VM and it complains that this "monitor" CPU flag is missing. Since QEMU/KVM cannot do anything about it, I was wondering if the ovirt dev team could ignore this flag please. Thanks

2 1

[VDSM] build-artifacts on s390x broken again (bad mock/dnf cache?)
by Nir Soffer 09 Mar '19

09 Mar '19

build-artifacts on s390x fail now with: [Errno 2] No such file or directory: '/var/cache/dnf/updates-07b7057ee4fded96/packages/systemd-238-12.git07f8cd5.fc28.s390x.rpm' Here are few example failures: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… This is going to block the change queue when we merge these patches, and we have 4.3 build soon. We had this issue few week ago, and Barak fixed this by deleting caches on the slaves. I hope this can be fixed quickly this time. Nir

1 0

ovirt-optimizer: subproject health check
by Sandro Bonazzola 08 Mar '19

08 Mar '19

Hi, oVirt Optimizer has been introduced in oVirt 3.5.2 back in 2014. Last update on master branch is dated November 2017 with version 0.15 The package has been shipped till oVirt 4.2.8 and it has not been shipped in oVirt 4.3 despite no official deprecation has been announced. According to Gerrit, current maintainers are Doron Fediuck and Martin Sivak. According to Bugzilla there is one bug open since February 2018: Bug 1545077 <https://bugzilla.redhat.com/show_bug.cgi?id=1545077> - cancel optimization button is enabled all the time which is currently untargeted, unassigned and without activity since March 2018. The project has feature / development documentation at https://ovirt.org/develop/release-management/features/sla/optaplanner.html but no official documentation. Can we declare this subproject retired from oVirt project and drop CI automation for it? Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

ovirt-scheduler-proxy: subproject health check
by Sandro Bonazzola 08 Mar '19

08 Mar '19

Hi, oVirt Scheduler Proxy has been introduced in oVirt 3.3.1 back in 2013. Last update on master branch is dated November 2017 with 0.1.8 released. The package has been shipped till oVirt 4.2.8 and it has not been shipped in oVirt 4.3 despite no official deprecation has been announced. According to Gerrit, current maintainers are Doron Fediuck and Martin Sivak. According to Bugzilla there are no open bugs or RFEs for this project. The project has no documentation available on oVirt website. Can we declare this subproject retired from oVirt project and drop CI automation for it? Thanks, -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

2 2

[ci test tool] how to start build-artifacts job using jenkins API?
by Nir Soffer 08 Mar '19

08 Mar '19

I'm trying to a write a tool that given a gerrit patch number, will: 1. Start build artifacts job (instead of adding "ci please build" comment)\ 2. Wait until the job is complete 3. Start OST with CUSTOM_REPOS using the build artifacts job from step 1 4. Wait until OST complets 5. Post a comment with the OST link to gerrit So testing patches will easy as: $ ci test 12345 Building rpms... Starting OST... OST succeeded, congratulations! I can start OST and watch it, but I could not find a way to start vdsm build-artifacts job. This starts a job: curl -i \ --user USERNAME:API_TOKEN \ -X POST \ https://jenkins.ovirt.org/job/vdsm_standard-check-patch/build But the fail quickly in the first "detecting" stage. https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… Clicking "Build Now" button on the job page fails in the same way. And this: curl -i \ --user USERNAME:API_TOKEN \ -X POST \ https://jenkins.ovirt.org/job/vdsm_standard-check-patch/buildWithParameters Fails with 500 Internal server error, with a huge traceback, hiding this expected message: java.lang.IllegalStateException: This build is not parameterized! Looks like the job need to be parametrized: - https://jenkins.ovirt.org/job/vdsm_standard-check-patch/configure - https://jenkins.ovirt.org/job/ovirt-system-tests_manual/configure I guess we can workaround this by posting "ci please build" comment on gerrit using gerrit API, and then waiting waiting an grabbing the build link from the comment added by jenkins, but I really don't want to go in this direction. I hope we can have a proper solution in the job configuration. Nir

1 1

[ OST Failure Report ] [ oVirt Master ] [ 07-03-2019 ] [ 008_basic_ui_sanity.initialize_firefox]
by Galit Rosenthal 07 Mar '19

07 Mar '19

Hi, we are failing basic suite master (vdsm) Can you please have a look at the issue? *Error Message:* Message: Error forwarding the new session new session request for webdriver should contain a location header or an 'application/json;charset=UTF-8' response body with the session ID. *from the [2]:* *08:22:04* + xmllint --format /dev/shm/ost/deployment-basic-suite-master/008_basic_ui_sanity.py.junit.xml*08:22:04* <?xml version="1.0" encoding="UTF-8"?>*08:22:04* <testsuite name="nosetests" tests="7" errors="1" failures="0" skip="0">*08:22:04* <testcase classname="008_basic_ui_sanity" name="init" time="0.001"/>*08:22:04* <testcase classname="008_basic_ui_sanity" name="start_grid" time="144.413">*08:22:04* <system-out><![CDATA[executing shell: docker ps*08:22:04* CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES*08:22:04* *08:22:04* *08:22:04* executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub*08:22:04* *08:22:04* Error response from daemon: No such container: grid_node_chrome*08:22:04* Error response from daemon: No such container: grid_node_firefox*08:22:04* Error response from daemon: No such container: selenium-hub*08:22:04* *08:22:04* executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub*08:22:04* *08:22:04* Error response from daemon: Cannot kill container grid_node_chrome: No such container: grid_node_chrome*08:22:04* Error response from daemon: Cannot kill container grid_node_firefox: No such container: grid_node_firefox*08:22:04* Error response from daemon: Cannot kill container selenium-hub: No such container: selenium-hub*08:22:04* *08:22:04* executing shell: docker ps*08:22:04* CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES*08:22:04* *08:22:04* *08:22:04* executing shell: docker network ls*08:22:04* NETWORK ID NAME DRIVER SCOPE*08:22:04* f710c07f46da bridge bridge local*08:22:04* 7d05935c9513 host host local*08:22:04* 415e4fd8022f none null local*08:22:04* *08:22:04* *08:22:04* executing shell: docker network rm grid*08:22:04* *08:22:04* Error response from daemon: network grid not found*08:22:04* *08:22:04* executing shell: docker network ls*08:22:04* NETWORK ID NAME DRIVER SCOPE*08:22:04* f710c07f46da bridge bridge local*08:22:04* 7d05935c9513 host host local*08:22:04* 415e4fd8022f none null local*08:22:04* *08:22:04* *08:22:04* executing shell: docker network ls*08:22:04* NETWORK ID NAME DRIVER SCOPE*08:22:04* f710c07f46da bridge bridge local*08:22:04* 7d05935c9513 host host local*08:22:04* 415e4fd8022f none null local*08:22:04* *08:22:04* *08:22:04* creating docker network for grid*08:22:04* executing shell: docker network create grid*08:22:04* 7406c15c141eccac968e5fba9f091d83831a29c92fc9f61b95bc4fba47f5e73f*08:22:04* *Global link to the CQ fail[3]* [1] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… [3] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… -- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)gmail.com T: 972-9-7692230 <https://red.ht/sig>

4 3

[ OST Failure Report ] [ oVirt Master ] [ 07-03-2019 ] [ / 004_basic_sanity.verify_glance_import]
by Galit Rosenthal 07 Mar '19

07 Mar '19

Hi, we are failing basic suite master ( ovirt-engine) It looks a problem in ovirt-engine Can you please have a look at the issue? Thanks, Galit Error [1]: 'NoneType' object has no attribute 'status' -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f5e9feb3ed8> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 1153, in <lambda> lambda: api.disks.get(disk_name).status.state == 'ok', AttributeError: 'NoneType' object has no attribute 'status' --------------------- >> end captured logging << ------------- US CQ of the ovirt-engine results can be found [2] We see the same error DS as well [3] [1] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chang… [3] https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/blue/rest/organi… -- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)gmail.com T: 972-9-7692230 <https://red.ht/sig>

5 9