- Devel - oVirt List Archives

test-repo_ovirt_experimental_master fails on jsonrpc errors
by Daniel Belenky 12 Jan '17

12 Jan '17

Hi all, The following job: test-repo_ovirt_experimental_master <http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_exper…> fails to pass the basic_suite. The job was triggered by this merge: https://gerrit.ovirt.org/#/c/69936/ to vdsm project. The error I suspect cause this issue: 2017-01-11 03:32:26,061-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc":"2.0","error":{"code":"192.168.201.2:990178830","message":"Vds timeout occured"},"id":null} 2017-01-11 03:32:26,067-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [57bc898] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM command failed: Message timeout which can be caused by communication issues 2017-01-11 03:32:26,069-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [57bc898] ERROR, GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='f92af272-934f-4327-9db0-afe353e6f61c', ignoreFailoverLimit='true'}), exception: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues, log id: 2f12b94a 2017-01-11 03:32:26,069-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [57bc898] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand.executeIrsBrokerCommand(GetStoragePoolInfoVDSCommand.java:32) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.lambda$executeVDSCommand$0(IrsBrokerCommand.java:95) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy.runInControlledConcurrency(IrsProxy.java:262) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand.executeVDSCommand(IrsBrokerCommand.java:92) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:408) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy.proceedStoragePoolStats(IrsProxy.java:348) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy.lambda$updatingTimerElapsed$0(IrsProxy.java:246) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy.runInControlledConcurrency(IrsProxy.java:262) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy.updatingTimerElapsed(IrsProxy.java:227) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] Attached is a zip file with all artifacts from Jenkins. The error I've mentioned above is found in: *exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovirt-engine/engine.log <http://post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi…>* Can some one advise? Thanks, -- *Daniel Belenky* *RHV DevOps* *Red Hat Israel*

4 5

ovirt master system tests fail
by Daniel Belenky 12 Jan '17

12 Jan '17

Hi all, test-repo ovirt experimental master job fails, and it seems that there is an issue with 'add_host' phase under the '*bootstrap*' suite. >From the logs, it seems that the suite was unable to fire up the host / something is wrong with host <error type="exceptions.RuntimeError" message="Host lago-basic-suite-master-host1 is in non operational state -------------------- >> begin captured logging << -------------------- lago.ssh: DEBUG: start task Get ssh client for lago-basic-suite-master-host0 lago.ssh: DEBUG: Still got 100 tries for lago-basic-suite-master-host0 lago.ssh: DEBUG: end task Get ssh client for lago-basic-suite-master-host0 lago.ssh: DEBUG: Running aab0eff8 on lago-basic-suite-master-host0: yum install -y iptables lago.ssh: DEBUG: Command aab0eff8 on lago-basic-suite-master-host0 returned with 0 lago.ssh: DEBUG: Command aab0eff8 on lago-basic-suite-master-host0 output: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: centos.host-engine.com * extras: linux.mirrors.es.net * updates: mirror.n5tech.com Package iptables-1.4.21-17.el7.x86_64 already installed and latest version Nothing to do lago.ssh: DEBUG: start task Get ssh client for lago-basic-suite-master-host1 lago.ssh: DEBUG: Still got 100 tries for lago-basic-suite-master-host1 lago.ssh: DEBUG: end task Get ssh client for lago-basic-suite-master-host1 lago.ssh: DEBUG: Running ab5c94f2 on lago-basic-suite-master-host1: yum install -y iptables lago.ssh: DEBUG: Command ab5c94f2 on lago-basic-suite-master-host1 returned with 0 lago.ssh: DEBUG: Command ab5c94f2 on lago-basic-suite-master-host1 output: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.n5tech.com * extras: ftp.osuosl.org * updates: mirrors.usc.edu Package iptables-1.4.21-17.el7.x86_64 already installed and latest version Nothing to do ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up at 0x322e938> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within res = func() File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 162, in _host_is_up raise RuntimeError('Host %s is in non operational state' % host.name()) RuntimeError: Host lago-basic-suite-master-host1 is in non operational state --------------------- >> end captured logging << ---------------------"> >From the engine.log, I found a timeout in the rpc call (but this error is seen on jobs that success too, so might not be relevant(?)) 2017-01-12 05:49:53,383-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-2) [76b0383f] Command 'PollVDSCommand(HostName = lago-basic-suite-master-host1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='40eb11ba-e6ac-478a-b8b1-73b7892ace65'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during rpc call 2017-01-12 05:49:53,383-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-2) [76b0383f] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Timeout during rpc call ... (the full error is very long, so I wont paste it here, its in the* engine.log*) 2017-01-12 05:49:58,291-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [30b2ca77] Timeout waiting for VDSM response: Internal timeout occured In the host's vdsm.log, there are some errors too: 2017-01-12 05:51:48,336 ERROR (jsonrpc/0) [storage.StorageDomainCache] looking for unfetched domain 380623d8-1e85-4831-9048-3d05932f3d3a (sdc:151) 2017-01-12 05:51:48,336 ERROR (jsonrpc/0) [storage.StorageDomainCache] looking for domain 380623d8-1e85-4831-9048-3d05932f3d3a (sdc:168) 2017-01-12 05:51:48,395 WARN (jsonrpc/0) [storage.LVM] lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "380623d8-1e85-4831-9048-3d05932f3d3a" not found', ' Cannot process volume group 380623d8-1e85-4831-9048-3d05932f3d3a'] (lvm:377) 2017-01-12 05:51:48,398 ERROR (jsonrpc/0) [storage.StorageDomainCache] domain 380623d8-1e85-4831-9048-3d05932f3d3a not found (sdc:157) Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'380623d8-1e85-4831-9048-3d05932f3d3a',) and 2017-01-12 05:53:45,375 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 43814, 0, 0) at 0x235a2d8>: unexpected eof (betterAsyncore:119) Link to Jenkins <http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_exper…> Can someone please take a look? Thanks, *Daniel Belenky* *RHV DevOps* *Red Hat Israel*

4 3

Introduction of cluster level 4.2 on master
by Martin Perina 12 Jan '17

12 Jan '17

Hi, we have just merged a series of patches to engine which introduces cluster level 4.2: 1. Adds cluster level 4.2 [1] 2. Sets version 4.2 to default data center and cluster for new installations [1] 2. Bumps database upgrade scripts to 4.2 [2] - from now on every database script merged to master has to start with 04_02 prefix Please rebase all your existing patches to master on top of those changes and rename db scripts accordingly. Also please be aware that if you want to add a host to a 4.2 cluster, you need quite recent VDSM version which includes patch [3] merged yesterday. Please let me know if you have any issues. Thanks Martin Perina [1] https://gerrit.ovirt.org/69757 [2] https://gerrit.ovirt.org/69758 [3] https://gerrit.ovirt.org/69759

1 0

[oVirt 4.1 Localization Question #1] "No VM configuration"
by Yuko Katabami 12 Jan '17

12 Jan '17

Hi oVirt developers, We have started translating 4.1 UI strings and would like to post our first question. File: CommonApplicationConstants Resource ID: brokenVmConfiguration String: No VM configuration Question: Could anyone tell me where in the UI it is located? Is this used as a part of an error message, or as a label? Kind regards, Yuko

2 2

ovirt-system-tests 4.1 is failling due to VmDisksResource class
by Daniel Belenky 12 Jan '17

12 Jan '17

Hi all, ovirt-system-tests are failing with the following error: ERROR [org.ovirt.engine.api.restapi.util.LinkHelper] (default task-1) [] Can't find relative path for class "org.ovirt.engine.api.resource.VmDisksResource", will return null The error began at 4/1. can someone take a look please? Attached all the logs under /var/log/ovirt-engine , the error I've mentioned above is seen in *engine.log*. Can someone please take a look? Thanks, -- *Daniel Belenky* *RHV DevOps* *Red Hat Israel*

3 2

(no subject)
by Daniel Belenky 12 Jan '17

12 Jan '17

On Thu, Jan 12, 2017 at 10:35 AM, Yaniv Kaul <ykaul(a)redhat.com> wrote: > > > On Thu, Jan 12, 2017 at 10:25 AM, Daniel Belenky <dbelenky(a)redhat.com> > wrote: > >> Hi all, >> >> ovirt-system-tests are failing with the following error: >> >> ERROR [org.ovirt.engine.api.restapi.util.LinkHelper] (default task-1) [] Can't find relative path for class "org.ovirt.engine.api.resource.VmDisksResource", will return null >> >> > This is a known issue ( https://bugzilla.redhat.com/ > show_bug.cgi?id=1410038 ) and is not a cause for failure. > > >> The error began at 4/1. >> > > This is really outdated. How did work yesterday? > It didn't. We just didn't get to this issue, but it's failing for a while. > >> can someone take a look please? >> > > What is the test that is failing? > basic_sanity is failing on vm_run Link <http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt_4.1_system-t…> to the failing job Thanks, *Daniel Belenky* *RHV DevOps* *Red Hat Israel*

1 0

planned Jenkins restart
by Evgheni Dereveanchin 11 Jan '17

11 Jan '17

Hi everyone, I'll be performing a planned Jenkins restart today. No new builds will be scheduled during this maintenance period. I will announce you once it is finished. Regards, Evgheni Dereveanchin

2 2

PatternFly upgrade - how to handle JS dependencies
by Vojtech Szocs 11 Jan '17

11 Jan '17

Hello, we'd like to upgrade the version of PatternFly (plus associated libraries, namely Bootstrap and jQuery) used in oVirt UI. Today, PatternFly stuff (PF + associated libraries) comes from `patternfly1` package hosted at Copr repo: https://copr.fedorainfracloud.org/coprs/patternfly/patternfly1/ Today, we require `patternfly1` as both Engine RPM build dependency and the Engine devel. env. dependency. To keep things simple, I'd like to propose the following approach: - create oVirt specific package, e.g. `ovirt-patternfly`, hosted at Copr, containing PatternFly + associated libraries (Bootstrap, jQuery) which are intended *specifically* for use by oVirt UI (hence the ovirt prefix) - discontinue maintenance of `patternfly1` package at Copr - keep the existing approach: require `ovirt-patternfly` as both the Engine build dependency and the Engine devel. env. dependency An alternative approach would be to introduce Node.js as Engine devel. env. dependency (use `npm install` to pull PF stuff), while using the existing ovirt-engine-{nodejs,nodejs-modules} packages for Engine RPM build. However, this alternative has some downsides, namely complication of devel. env. just to fetch the PF stuff. I don't like the idea of complicating the existing devel. env. just for the sake of fetching some 3rd party libs. I'm wondering what others think about this. Regards, Vojtech

4 10

Fwd: [oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4658 - FAILURE!
by Piotr Kliczewski 11 Jan '17

11 Jan '17

Nir, I saw similar issue on when I run system tests on my machine. Here is piece of log:: 2017-01-10 09:44:56,442 WARN (jsonrpc/3) [storage.HSM] getPV failed for guid: 3600140527d13be6be704ff98fd8838c2 (hsm:1973) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 1970, in _getDeviceList pv = lvm.getPV(guid) File "/usr/share/vdsm/storage/lvm.py", line 856, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'3600140527d13be6be704ff98fd8838c2',)" 2017-01-10 09:44:57,493 ERROR (jsonrpc/4) [storage.StorageDomainCache] domain d67c721e-d1f2-484f-a647-13a82db90b33 not found (sdc:157) Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 155, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 185, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'd67c721e-d1f2-484f-a647-13a82db90b33',) Can you please take a look why are we seeing it? Thanks, Piotr ---------- Forwarded message ---------- From: <jenkins(a)jenkins.phx.ovirt.org> Date: Tue, Jan 10, 2017 at 4:08 PM Subject: [oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4658 - FAILURE! To: infra(a)ovirt.org Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4658/, Build Number: 4658, Build Status: FAILURE _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

2 2

suspend_resume_vm fail on master experimental
by Daniel Belenky 11 Jan '17

11 Jan '17

Hi all, *test-repo_ovirt_experimental_master* (link to Jenkins <http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_exper…>) job failed on basic_sanity scenario. The job was triggered by https://gerrit.ovirt.org/#/c/69845/ >From looking at the logs, it seems that the reason is *VDSM*. In the VDSM log, i see the following error: 2017-01-09 16:47:41,331 ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL error receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1', 34942, 0, 0) at 0x36b95f0>: unexpected eof (betterAsyncore:119) Also, when looking at the MOM logs, I see the the following: 2017-01-09 16:43:39,508 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused I've attached the full VDSM logs here in a zip file. Can anyone please assist? Thanks, -- *Daniel Belenky* *RHV DevOps* *Red Hat Israel*

6 13