[ovirt-devel] [OST Failure Report][oVirt master][2017-04-11] add_secondary_storage_domains
Michal Skrivanek
michal.skrivanek at redhat.com
Wed Apr 12 09:44:26 UTC 2017
> On 12 Apr 2017, at 09:17, Sandro Bonazzola <sbonazzo at redhat.com> wrote:
>
>
>
> On Wed, Apr 12, 2017 at 9:03 AM, Michal Skrivanek <michal.skrivanek at redhat.com <mailto:michal.skrivanek at redhat.com>> wrote:
>
>> On 12 Apr 2017, at 08:37, Yaniv Kaul <ykaul at redhat.com <mailto:ykaul at redhat.com>> wrote:
>>
>>
>>
>> On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <bkorren at redhat.com <mailto:bkorren at redhat.com>> wrote:
>> This seems to have been the Secodary SD issue I reported on Sunday:
>> http://lists.ovirt.org/pipermail/devel/2017-April/030139.html <http://lists.ovirt.org/pipermail/devel/2017-April/030139.html>
>>
>> That was up until build #6307 where AddHost started failing:
>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307 <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307>
>>
>> That's an entirely different game:
>> 2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
>> 2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) [3497aa05] Exception: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
>
> this one started in http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307/>
>
> there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is what seems to be mismatched
> Piotr/Sandro, are all artifacts published correctly?
>
> Please define published correctly.
> Since the publisher job is not used anymore to publish artifacts I've no list of what gets published.
Well, Im not sure who’s handling that exactly, but vdsm-jsonrpc-java is getting into OST runs despite the OST failing since Saturday. That should not happen
btw, in the meantime, me and Tomas are working on narrowing down the original failure.
Thanks,
michal
> I can only check repository content:
>
> cockpit-ovirt-dashboard-0:0.11.0-0.0.2.el7.centos.noarch
> cockpit-ovirt-uiplugin-0:0.11.0-0.0.2.el7.centos.noarch
> imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch
> ioprocess-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64
> ioprocess-debuginfo-0:0.17.0-1.201702282323.gite82786d.el7.centos.x86_64
> java-ovirt-engine-sdk4-0:4.1.0-0.1.20170331gitdcf3f4d.el7.centos.noarch
> libcacard-devel-ev-10:2.3.0-31.el7_2.21.1.x86_64
> libcacard-ev-10:2.3.0-31.el7_2.21.1.x86_64
> libcacard-tools-ev-10:2.3.0-31.el7_2.21.1.x86_64
> mom-0:0.5.9-0.0.master.el7.centos.noarch
> nodectl-0:4.1-999.20160902.0.el7.noarch
> otopi-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch
> otopi-debug-plugins-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch
> otopi-devtools-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch
> otopi-java-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch
> otopi-javadoc-0:1.7.0-0.0.master.20170405141623.git0166f14.el7.centos.noarch
> ovirt-engine-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-appliance-0:4.2-20170408.1.el7.centos.noarch
> ovirt-engine-backend-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-cli-0:3.6.9.3-0.1.20170116.gitad0561a.el7.centos.noarch
> ovirt-engine-dashboard-0:1.2.0-0.5.20170331git8568fc5.el7.centos.noarch
> ovirt-engine-dbscripts-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-dwh-0:4.2.0-0.0.master.20170403125608.el7.centos.noarch
> ovirt-engine-dwh-setup-0:4.2.0-0.0.master.20170403125608.el7.centos.noarch
> ovirt-engine-extension-aaa-jdbc-0:1.1.5-0.0.master.git2a03d42.el7.centos.noarch
> ovirt-engine-extension-aaa-ldap-0:1.3.2-0.0.master.gitabe7725.el7.centos.noarch
> ovirt-engine-extension-aaa-ldap-setup-0:1.3.2-0.0.master.gitabe7725.el7.centos.noarch
> ovirt-engine-extension-aaa-misc-0:1.0.2-0.0.master.git9430be4.el7.centos.noarch
> ovirt-engine-extension-logger-log4j-0:1.0.3-0.0.master.el7.centos.noarch
> ovirt-engine-extensions-api-impl-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-extensions-api-impl-javadoc-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-hosts-ansible-inventory-0:4.2.0-0.0.master.20170302072348.git4674298.el7.centos.noarch
> ovirt-engine-lib-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-metrics-0:1.0.3-0.1.master.20170406102316.el7.centos.noarch
> ovirt-engine-nodejs-0:6.9.4-5.el7.centos.x86_64
> ovirt-engine-nodejs-modules-0:1.0.4-5.el7.centos.x86_64
> ovirt-engine-nodejs-modules-0:1.0.7-1.el7.centos.noarch
> ovirt-engine-phantomjs-0:2.1.1-1.el7.centos.x86_64
> ovirt-engine-phantomjs-debuginfo-0:2.1.1-1.el7.centos.x86_64
> ovirt-engine-restapi-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-sdk-java-0:3.6.10.0-1.20170215git67dcbcd.el7.centos.noarch
> ovirt-engine-sdk-java-javadoc-0:3.6.10.0-1.20170215git67dcbcd.el7.centos.noarch
> ovirt-engine-sdk-python-0:3.6.9.2-0.1.20161204.gite99bbd1.el7.centos.noarch
> ovirt-engine-setup-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-setup-base-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-setup-plugin-dockerc-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-setup-plugin-live-0:4.2.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch
> ovirt-engine-setup-plugin-ovirt-engine-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-setup-plugin-ovirt-engine-common-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-setup-plugin-vmconsole-proxy-helper-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-setup-plugin-websocket-proxy-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-tools-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-tools-backup-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-userportal-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-userportal-debuginfo-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-vmconsole-proxy-helper-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-webadmin-portal-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-webadmin-portal-debuginfo-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-websocket-proxy-0:4.2.0-0.0.master.20170408115943.gitea6bd62.el7.centos.noarch
> ovirt-engine-wildfly-0:10.1.0-1.el7.x86_64
> ovirt-engine-wildfly-overlay-0:10.0.0-1.el7.noarch
> ovirt-engine-yarn-0:0.19.1-2.el7.centos.x86_64
> ovirt-engine-yarn-0:0.19.1-3.el7.centos.noarch
> ovirt-guest-agent-common-0:1.0.13-1.el7.centos.noarch
> ovirt-guest-agent-debuginfo-0:1.0.13-1.el7.centos.x86_64
> ovirt-guest-agent-gdm-plugin-0:1.0.13-1.el7.centos.noarch
> ovirt-guest-agent-pam-module-0:1.0.13-1.el7.centos.x86_64
> ovirt-host-deploy-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.noarch
> ovirt-host-deploy-java-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.noarch
> ovirt-host-deploy-javadoc-0:1.7.0-0.0.master.20170407114427.gitb41f5b9.el7.centos.noarch
> ovirt-hosted-engine-ha-0:2.2.0-0.0.master.20170404202238.20170404202234.git68258f1.el7.centos.noarch
> ovirt-hosted-engine-setup-0:2.2.0-0.0.master.20170330072130.git8a05e04.el7.centos.noarch
> ovirt-image-uploader-0:4.1.0-0.0.master.20160830110830.gitad0be34.el7.centos.noarch
> ovirt-imageio-common-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch
> ovirt-imageio-daemon-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch
> ovirt-imageio-proxy-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch
> ovirt-imageio-proxy-setup-0:1.0.0-0.201703211442.gitfa6a59d.el7.centos.noarch
> ovirt-iso-uploader-0:4.1.0-0.0.master.20160909154152.git14502bd.el7.centos.noarch
> ovirt-jboss-modules-maven-plugin-0:2.0.2-0.1.20160715.gitec01507.el7.centos.noarch
> ovirt-jboss-modules-maven-plugin-javadoc-0:2.0.2-0.1.20160715.gitec01507.el7.centos.noarch
> ovirt-js-dependencies-0:0.0.3-2.el7.centos.x86_64
> ovirt-js-dependencies-0:0.0.3-4.el7.centos.noarch
> ovirt-live-artwork-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch
> ovirt-live-artwork-gnome-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch
> ovirt-live-artwork-mate-0:1.1.0-0.0.master.20170331075633.git8b425a3.el7.centos.noarch
> ovirt-log-collector-0:4.1.0-0.0.master.20170405134355.gitd0eaad4.el7.centos.noarch
> ovirt-log-collector-analyzer-0:4.1.0-0.0.master.20170405134355.gitd0eaad4.el7.centos.noarch
> ovirt-node-ng-docs-0:4.1.0-0.20170408.0.el7.noarch
> ovirt-node-ng-image-0:4.2.0-0.3.master.20170406003036.git2f43a3c.el7.centos.noarch
> ovirt-node-ng-image-update-0:4.2.0-0.3.master.20170406003036.git2f43a3c.el7.centos.noarch
> ovirt-node-ng-image-update-placeholder-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch
> ovirt-node-ng-nodectl-0:4.1.0-0.20170408.0.el7.noarch
> ovirt-node-ng-tools-0:4.1.0-0.20170408.0.el7.noarch
> ovirt-optimizer-0:0.15-0.176.201702021258.el7.centos.noarch
> ovirt-optimizer-dependencies-0:0.15-0.176.201702021258.el7.centos.noarch
> ovirt-optimizer-jboss-0:0.15-0.176.201702021258.el7.centos.noarch
> ovirt-optimizer-ui-0:0.15-0.176.201702021258.el7.centos.noarch
> ovirt-provider-ovn-0:1.1-2.20170407114314.git8f0d7ee.el7.centos.noarch
> ovirt-provider-ovn-driver-0:1.1-2.20170407114314.git8f0d7ee.el7.centos.noarch
> ovirt-release-host-node-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch
> ovirt-release-master-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch
> ovirt-release-master-experimental-0:4.2.0-0.3.master.20170408002716.git2f43a3c.el7.centos.noarch
> ovirt-setup-lib-0:1.1.0-1.20161219151842.git402fec4.el7.centos.noarch
> ovirt-vmconsole-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.noarch
> ovirt-vmconsole-host-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.noarch
> ovirt-vmconsole-proxy-0:1.0.4-0.0.master.20170301155209.git50f0014.el7.centos.noarch
> ovirt-web-ui-0:0.1.2-1.el7.centos.x86_64
> python-imgbased-0:0.9.999-0.201704061527gita27110c.el7.centos.noarch
> python-ioprocess-0:0.17.0-1.201702282324.gite82786d.el7.centos.noarch
> python-ovirt-engine-sdk4-0:4.2.0-1.a0.20170331gitd6b662b.el7.centos.x86_64
> python-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.a0.20170331gitd6b662b.el7.centos.x86_64
> qemu-img-ev-10:2.6.0-28.el7_3.6.1.x86_64
> qemu-kvm-common-ev-10:2.6.0-28.el7_3.6.1.x86_64
> qemu-kvm-ev-10:2.6.0-28.el7_3.6.1.x86_64
> qemu-kvm-ev-debuginfo-10:2.6.0-28.el7_3.6.1.x86_64
> qemu-kvm-tools-ev-10:2.6.0-28.el7_3.6.1.x86_64
> rubygem-ovirt-engine-sdk4-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x86_64
> rubygem-ovirt-engine-sdk4-debuginfo-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x86_64
> rubygem-ovirt-engine-sdk4-doc-0:4.2.0-1.alpha0.20170407gite402de7.el7.centos.x86_64
> unboundid-ldapsdk-0:3.2.0-1.el7.noarch
> unboundid-ldapsdk-javadoc-0:3.2.0-1.el7.noarch
> vdsm-0:4.20.0-612.git378d837.el7.centos.x86_64
> vdsm-api-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-client-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-gluster-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-allocate_net-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-checkimages-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-checkips-0:4.20.0-612.git378d837.el7.centos.x86_64
> vdsm-hook-diskunmap-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-ethtool-options-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-extnet-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-extra-ipv4-addrs-0:4.20.0-612.git378d837.el7.centos.x86_64
> vdsm-hook-fakesriov-0:4.20.0-612.git378d837.el7.centos.x86_64
> vdsm-hook-fakevmstats-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-faqemu-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-fcoe-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-fileinject-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-floppy-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-hostusb-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-httpsisoboot-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-hugepages-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-ipv6-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-isolatedprivatevlan-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-localdisk-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-macbind-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-macspoof-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-nestedvt-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-noipspoof-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-numa-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-openstacknet-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-pincpu-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-promisc-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-qemucmdline-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-qos-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-scratchpad-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-smbios-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-spiceoptions-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-vhostmd-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-vmdisk-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-vmfex-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-hook-vmfex-dev-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-infra-0:4.18.999-212.git2556c05.el7.centos.noarch
> vdsm-jsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-jsonrpc-java-0:1.4.5-1.20170330074257.gitbef06c7.el7.centos.noarch
> vdsm-jsonrpc-java-javadoc-0:1.4.5-1.20170330074257.gitbef06c7.el7.centos.noarch
> vdsm-python-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-tests-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-xmlrpc-0:4.20.0-612.git378d837.el7.centos.noarch
> vdsm-yajsonrpc-0:4.20.0-612.git378d837.el7.centos.noarch
>
>
>
>
>
>
> Thanks,
> michal
>
>> org/apache/commons/lang/StringUtils
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.8.0_121]
>> at java.util.concurrent.FutureTask.get(FutureTask.java:206) [rt.jar:1.8.0_121]
>> at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:69) [vdsbroker.jar:]
>> at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) [bll.jar:]
>> at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) [bll.jar:]
>>
>> ...
>> Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
>> at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message.withCorrelationId(Message.java:75) [vdsm-jsonrpc-java-client.jar:]
>> at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SSLStompClient.java:85) [vdsm-jsonrpc-java-client.jar:]
>> at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) [vdsm-jsonrpc-java-client.jar:]
>> at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91) [vdsbroker.jar:]
>> at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundPoll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:]
>> at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:484) [vdsbroker.jar:]
>> at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:473) [vdsbroker.jar:]
>>
>>
>>
>>
>>
>> On 11 April 2017 at 18:12, Piotr Kliczewski <piotr.kliczewski at gmail.com <mailto:piotr.kliczewski at gmail.com>> wrote:
>> I looks like SD activate failed on the engine side and the spm connection was gently closed by the engine:
>>
>> 2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st <http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 09:10:24 EDT 2017
>> 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to non-operational: null
>> 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.storage.pool.RefreshPoolSingleAsyncOperation] (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to non-operational: null
>> 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.Se <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1b611f2-67fb-4acb-8261-07133134e0e7 Type: VDS
>> 2017-04-11 09:10:24,909-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host0, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239
>> 2017-04-11 09:10:24,919-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239
>> 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] Compiled stored procedure. Call string is [{call getvmsrunningonvds(?)}]
>> 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled
>> 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host state to Non-Operational.
>> 2017-04-11 09:10:24,985-04 DEBUG [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org <http://default.org/>.ovirt.engine.core.vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no unfired trigger.
>> 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.Se <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 87a5f691-ea5c-4692-a4ab-52dad0dafe34 Type: VDS
>> 2017-04-11 09:10:25,039-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b
>>
>> .....
>>
>> 2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, SpmStopVDSCommand, log id: 62f82cfb
>> 2017-04-11 09:10:25,366-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default task-29) [2ffa5981] UNSUBSCRIBE
>> id:4cd59c67-1b57-4456-9a16-52a72ac6f689
>>
>> \00
>> 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message sent: UNSUBSCRIBE
>>
>>
>>
>> On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <sbonazzo at redhat.com <mailto:sbonazzo at redhat.com>> wrote:
>> Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303 <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6303>
>>
>> OST is failing in the last 50 builds
>>
>>
>> 00:18:22.090 [basic_suit_el7] # add_secondary_storage_domains:
>> 00:18:22.090 [basic_suit_el7] Error while running thread
>> 00:18:22.090 [basic_suit_el7] Traceback (most recent call last):
>> 00:18:22.090 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
>> 00:18:22.090 [basic_suit_el7] queue.put({'return': func()})
>> 00:18:22.090 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 427, in add_nfs_storage_domain
>> 00:18:22.090 [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH, nfs_version='v4_2')
>> 00:18:22.091 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain
>> 00:18:22.091 [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
>> 00:18:22.091 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4
>> 00:18:22.091 [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),
>> 00:18:22.091 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4
>> 00:18:22.091 [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))
>> 00:18:22.091 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4
>> 00:18:22.091 [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
>> 00:18:22.092 [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc
>> 00:18:22.092 [basic_suit_el7] Error while running thread
>> 00:18:22.092 [basic_suit_el7] Traceback (most recent call last):
>> 00:18:22.092 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
>> 00:18:22.092 [basic_suit_el7] queue.put({'return': func()})
>> 00:18:22.092 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 583, in add_iso_storage_domain
>> 00:18:22.092 [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, SD_ISO_PATH, sd_format='v1', sd_type='iso', nfs_version='v3')
>> 00:18:22.092 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain
>> 00:18:22.092 [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
>> 00:18:22.093 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4
>> 00:18:22.093 [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),
>> 00:18:22.093 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4
>> 00:18:22.093 [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))
>> 00:18:22.093 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4
>> 00:18:22.093 [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
>> 00:18:22.093 [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc
>> 00:18:22.094 [basic_suit_el7] Error while running thread
>> 00:18:22.094 [basic_suit_el7] Traceback (most recent call last):
>> 00:18:22.094 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
>> 00:18:22.094 [basic_suit_el7] queue.put({'return': func()})
>> 00:18:22.094 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 587, in add_templates_storage_domain
>> 00:18:22.094 [basic_suit_el7] add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', sd_type='export', nfs_version='v4_1')
>> 00:18:22.094 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain
>> 00:18:22.094 [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
>> 00:18:22.094 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4
>> 00:18:22.095 [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),
>> 00:18:22.095 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4
>> 00:18:22.095 [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))
>> 00:18:22.095 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4
>> 00:18:22.095 [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
>> 00:18:22.095 [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc
>> 00:18:22.095 [basic_suit_el7] Error while running thread
>> 00:18:22.095 [basic_suit_el7] Traceback (most recent call last):
>> 00:18:22.096 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
>> 00:18:22.096 [basic_suit_el7] queue.put({'return': func()})
>> 00:18:22.096 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 436, in add_second_nfs_storage_domain
>> 00:18:22.096 [basic_suit_el7] SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)
>> 00:18:22.096 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 441, in add_generic_nfs_storage_domain
>> 00:18:22.096 [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)
>> 00:18:22.096 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 490, in add_generic_nfs_storage_domain_4
>> 00:18:22.096 [basic_suit_el7] host=_random_host_from_dc_4(api, DC_NAME),
>> 00:18:22.096 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 106, in _random_host_from_dc_4
>> 00:18:22.097 [basic_suit_el7] return random.choice(_hosts_in_dc_4(api, dc_name))
>> 00:18:22.097 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 100, in _hosts_in_dc_4
>> 00:18:22.097 [basic_suit_el7] raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
>> 00:18:22.097 [basic_suit_el7] RuntimeError: Could not find hosts that are up in DC test-dc
>> 00:18:22.097 [basic_suit_el7] Error while running thread
>> 00:18:22.097 [basic_suit_el7] Traceback (most recent call last):
>> 00:18:22.097 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
>> 00:18:22.098 [basic_suit_el7] queue.put({'return': func()})
>> 00:18:22.098 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 803, in import_template_from_glance
>> 00:18:22.098 [basic_suit_el7] generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, image_ext='_glance_template', as_template=True)
>> 00:18:22.098 [basic_suit_el7] File "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 637, in generic_import_from_glance
>> 00:18:22.098 [basic_suit_el7] target_image.import_image(import_action)
>> 00:18:22.098 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26017, in import_image
>> 00:18:22.098 [basic_suit_el7] headers={"Correlation-Id":correlation_id}
>> 00:18:22.098 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
>> 00:18:22.098 [basic_suit_el7] persistent_auth=self.__persistent_auth
>> 00:18:22.099 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
>> 00:18:22.099 [basic_suit_el7] persistent_auth)
>> 00:18:22.099 [basic_suit_el7] File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
>> 00:18:22.099 [basic_suit_el7] raise errors.RequestError(response_code, response_reason, response_body)
>> 00:18:22.099 [basic_suit_el7] RequestError:
>> 00:18:22.099 [basic_suit_el7] status: 400
>> 00:18:22.099 [basic_suit_el7] reason: Bad Request
>> 00:18:22.099 [basic_suit_el7] detail: Cannot import Virtual Disk: Storage Domain cannot be accessed.
>> 00:18:22.099 [basic_suit_el7] -Please check that at least one Host is operational and Data Center state is up.
>>
>> --
>> SANDRO BONAZZOLA
>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org <mailto:Devel at ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org <mailto:Devel at ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel>
>>
>>
>>
>> --
>> Barak Korren
>> bkorren at redhat.com <mailto:bkorren at redhat.com>
>> RHCE, RHCi, RHV-DevOps Team
>> https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org <mailto:Devel at ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org <mailto:Devel at ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel>
>
>
>
> --
> SANDRO BONAZZOLA
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170412/5ca300e5/attachment-0001.html>
More information about the Devel
mailing list