- Users - oVirt List Archives

Issues post-upgrade from 4.2.8 to 4.3.2
by Vrgotic, Marko 21 Apr '19

21 Apr '19

Dear oVirt team, We have 3 Hosts HA Cluster with SHE engine. Storage is NetApp. Recently we have executed an upgrade from 4.2.8 to 4.3.2. Since then we are seeing strange behavior when trying to put Hosts to Maintenance or manually Migrate VMs between Hosts. The most repeated WARN/ERROR I am observing is following: The UI is constantly throwing an exception when trying to put HA host to maintenance or increase the SPM level. The repeated log line from engine.log I see, with regards or failing to migrate all hosts is: 2019-04-18 13:05:30,760Z WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [63d33dd1] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-02,$filterName Memory,$availableMem 3824,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-01,$filterName Memory,$availableMem 0,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-02,$filterName Memory,$availableMem 3824,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-01,$filterName Memory,$availableMem 0,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL But there is more than enough. Also from UI.log: 2019-04-18 14:05:09,007Z ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-114) [] Permutation name: 0D2DB7A91B469CC36C64386E5632FAC5 2019-04-18 14:05:09,007Z ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-114) [] Uncaught exception: com.google.gwt.event.shared.Umbrell aException: Exception caught: (TypeError) : oab(...) is null at java.lang.Throwable.Throwable(Throwable.java:70) [rt.jar:1.8.0_201] at java.lang.RuntimeException.RuntimeException(RuntimeException.java:32) [rt.jar:1.8.0_201] at com.google.web.bindery.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:64) [gwt-servlet.jar:] at com.google.gwt.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:25) [gwt-servlet.jar:] at com.google.gwt.event.shared.HandlerManager.$fireEvent(HandlerManager.java:117) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.$fireEvent(Widget.java:127) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.fireEvent(Widget.java:127) [gwt-servlet.jar:] at com.google.gwt.event.dom.client.DomEvent.fireNativeEvent(DomEvent.java:110) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.$onBrowserEvent(Widget.java:163) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.onBrowserEvent(Widget.java:163) [gwt-servlet.jar:] at com.google.gwt.user.client.DOM.dispatchEvent(DOM.java:1415) [gwt-servlet.jar:] at com.google.gwt.user.client.impl.DOMImplStandard.dispatchEvent(DOMImplStandard.java:312) [gwt-servlet.jar:] at com.google.gwt.core.client.impl.Impl.apply(Impl.java:236) [gwt-servlet.jar:] at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:] at Unknown.Su/<(https://ovirt-staging-engine.avinity.tv/ovirt-engine/webadmin/?locale=en_US… line 9 > scriptElement) at Unknown.anonymous(Unknown) Caused by: com.google.gwt.core.client.JavaScriptException: (TypeError) : oab(...) is null at org.ovirt.engine.ui.uicommonweb.models.clusters.ClusterGuideModel.$onAddHost(ClusterGuideModel.java:533) at org.ovirt.engine.ui.uicommonweb.models.clusters.ClusterGuideModel.executeCommand(ClusterGuideModel.java:617) at org.ovirt.engine.ui.uicommonweb.UICommand.$execute(UICommand.java:163) at org.ovirt.engine.ui.common.presenter.AbstractModelBoundPopupPresenterWidget.$lambda$4(AbstractModelBoundPopupPresenterWidget.java:306) at org.ovirt.engine.ui.common.presenter.AbstractModelBoundPopupPresenterWidget$lambda$4$Type.onClick(AbstractModelBoundPopupPresenterWidget.java:306) at com.google.gwt.event.dom.client.ClickEvent.dispatch(ClickEvent.java:55) [gwt-servlet.jar:] at com.google.gwt.event.shared.GwtEvent.dispatch(GwtEvent.java:76) [gwt-servlet.jar:] at com.google.web.bindery.event.shared.SimpleEventBus.$doFire(SimpleEventBus.java:173) [gwt-servlet.jar:] ... 12 more Can you give me some advice on how to proceed? This started happening since upgrade. Kindly awaiting your reply, Marko Vrgotic

1 0

VM Snapshots not erasable and not bootable
by Jonathan Baecker 21 Apr '19

21 Apr '19

Hello, I make automatically backups of my VMs and last night there was making some new one. But somehow ovirt could not delete the snapshots anymore, in the log it show that it tried the hole day to delete them but they had to wait until the merge command was done. In the evening the host was totally crashed and started again. Now I can not delete the snapshots manually and I can also not start the VMs anymore. In the web interface I get the message: VM timetrack is down with error. Exit message: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'serial': 'fd3b80fd-49ad-44ac-9efd-1328300582cd', 'index': 0, 'iface': 'scsi', 'apparentsize': '1572864', 'specParams': {}, 'cache': 'none', 'imageID': 'fd3b80fd-49ad-44ac-9efd-1328300582cd', 'truesize': '229888', 'type': 'disk', 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'reqsize': '0', 'format': 'cow', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'device': 'disk', 'path': '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fd3b80fd-49ad-44ac-9efd-1328300582cd/47c0f42e-8bda-4e3f-8337-870899238788', 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': '47c0f42e-8bda-4e3f-8337-870899238788', 'diskType': 'file', 'alias': 'ua-fd3b80fd-49ad-44ac-9efd-1328300582cd', 'discard': False}. When I check the path permission is correct and there are also files in it. Is there any ways to fix that? Or to prevent this issue in the future? In the attachment I send also the engine.log Regards Jonathan

2 10

Re: Ovirt gluster arbiter within hosted VM
by Strahil 20 Apr '19

20 Apr '19

With GlusterD2 , you can use thin arbiter - which can be deployed in the cloud. Do not try to setup regular arbiter far away from the data bicks or yoir performance will be awful. Best Regards, Strahil NikolovOn Apr 19, 2019 13:28, Alex K <rightkicktech(a)gmail.com> wrote: > > > > On Fri, Apr 19, 2019 at 1:14 PM Scott Worthington <scott.c.worthington(a)gmail.com> wrote: >> >> And where, magically, is that node's storage going to live? > > In the disk of the VM. Let me know if I need to clarify further. >> >> >> You can't fake a proper setup of gluster. > > Not trying to fake it. Trying to find a solution with the available options. >> >> >> On Fri, Apr 19, 2019, 5:00 AM Alex K <rightkicktech(a)gmail.com> wrote: >>> >>> Hi all, >>> >>> I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes. >>> >>> What do you think about this setup in regards to stability and performance? >>> I am running ovirt 4.2. >>> >>> Thanx, >>> Alex >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTDOAMBKVE5BP…

2 1

Re: Ovirt gluster arbiter within hosted VM
by Strahil 20 Apr '19

20 Apr '19

As long as the VM is not hosted by those 2 hyperconverged nodes - there should be no problem. Another option is a small machine with a single SSD. I'm using Lenovo Tiny M-series as an arbiter. Best Regards, Strahil NikolovOn Apr 19, 2019 12:59, Alex K <rightkicktech(a)gmail.com> wrote: > > Hi all, > > I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes. > > What do you think about this setup in regards to stability and performance? > I am running ovirt 4.2. > > Thanx, > Alex

1 0

Replica 3 distribute-replicated - data placement and fault tolerance
by Leo David 19 Apr '19

19 Apr '19

Hello Everyone, I did some fio performance tests on a particular vm and I have noticed things that I do not understand regarding how data is placed along the bricks. I am sure this is a lack of knowledge, but I would really appreciate any help in understanding this, I did a bit of research on the internet, but just could't find something relevant. I have one replica 3 distributed-replicated arbitrated volume, across 18 bricks ( 9 nodes, 2 jbods per node ). The volume was created as: node1 - brick1, node-2, brick1,.....node9-brick1, node1 - brick2, node-2, brick2,.....node9-brick2 As far as i've understood, under the hood there are sets of 3 times replicated data ( subvolumes ) which are assigned to the first 3 bricks, next set of replicated data to the next set of 3 bricks, and so on.. Now, I have this testing vm runign one node one. When I've started the fio test, i've noticed incresed gluster traffic from node 1 to node2,4 and 5 So I assumed that the vm disk is data resides on a subvolume allocated to bricks from these hosts. Then I have migrated the vm on node 2, and did the same test. Now the increased treffic is generated from node2 to node1, 4, and 5.. What I do not understand is: - why gluster client ( ovirt host ) is sending data to 3 bricks if the volume is arbitrated - shouldn't it send only to 2 of them ? - why are there 4 brick implicated in this subvolume - what would it be the fault tolerance level in this setup ,ie: how many hosts can I take down and still having the volume serving IO requests; can they be random ? I am sorry for my lack of knowledge, I am just trying to understand what is happening so I can deploy a decent proper setup of an hci environment. Thank you, Leo -- Best regards, Leo David

1 0

Re: Ovirt gluster arbiter within hosted VM
by Alex K 19 Apr '19

19 Apr '19

On Fri, Apr 19, 2019 at 1:14 PM Scott Worthington < scott.c.worthington(a)gmail.com> wrote: > And where, magically, is that node's storage going to live? > In the disk of the VM. Let me know if I need to clarify further. > > You can't fake a proper setup of gluster. > Not trying to fake it. Trying to find a solution with the available options. > > On Fri, Apr 19, 2019, 5:00 AM Alex K <rightkicktech(a)gmail.com> wrote: > >> Hi all, >> >> I have a two node hyper-converged setup which are causing me split-brains >> when network issues are encountered. Since I cannot add a third hardware >> node, I was thinking to add a dedicated guest VM hosted in same >> hyper-converged cluster which would do the arbiter for the volumes. >> >> What do you think about this setup in regards to stability and >> performance? >> I am running ovirt 4.2. >> >> Thanx, >> Alex >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTDOAMBKVE5BP… >> >

1 0

Ovirt gluster arbiter within hosted VM
by Alex K 19 Apr '19

19 Apr '19

Hi all, I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes. What do you think about this setup in regards to stability and performance? I am running ovirt 4.2. Thanx, Alex

1 0

Ovirt-engine self hostd
by Dennis Perfors 19 Apr '19

19 Apr '19

Hi, We have build Ovirt-engine self hosted. We have created 2 vlans 1 services and 1 student. This works good. Services vlan 192.168.60.0/24 gateway ip 192.168.60.1 on this card Student vlan 192.168.10.0/24 gateway ip 192.168.10.1 on this network card Everything seems to be oke, but the probem is on the vlan. On the node 192.168.60.3, there is no internet connection when we do yum update -y But there is no internet connection on the host. I cant do yum update -y or do anything else. Here more information: ^Chttp://mirror.cj2.nl/centos/7.6.1810/extras/x86_64/repodata/repomd.xml: [Errno 14] curl#56 - "Callback aborted" Trying other mirror. ^Chttp://mirror.sitbv.nl/centos/7.6.1810/extras/x86_64/repodata/repomd.xml: [Errno 14] curl#56 - "Callback aborted" Trying other mirror. http://mirror.serverius.net/centos/7.6.1810/extras/x86_64/repodata/repomd.x…: [Errno 14] curl#7 - "Failed to connect to 2a03:3f40:1::15: Network is unreachable" [root@bastion ~]# ping nu.nl PING nu.nl (52.85.140.48) 56(84) bytes of data. 64 bytes from server-52-85-140-48.man50.r.cloudfront.net (52.85.140.48): icmp_seq=1 ttl=241 time=12.8 ms ^C [root@bastion ~]# ping alibabia.com PING alibabia.com (137.175.26.243) 56(84) bytes of data. 64 bytes from 137.175.26.243 (137.175.26.243): icmp_seq=1 ttl=115 time=144 ms 64 bytes from 137.175.26.243 (137.175.26.243): icmp_seq=2 ttl=115 time=144 ms ^C ipv4.method: manual ipv4.dns: 1.1.1.1 ipv4.dns-search: -- ipv4.dns-options: "" ipv4.dns-priority: 0 ipv4.addresses: 192.168.60.3/24 ipv4.gateway: 192.168.60.1 ipv4.routes: -- ipv4.route-metric: -1 ipv4.route-table: 0 (unspec) ipv4.ignore-auto-routes: no ipv4.ignore-auto-dns: no public (active) target: default icmp-block-inversion: no interfaces: ovirtmgmt sources: services: ssh dhcpv6-client ovirt-postgres ovirt-https ovn-central-firewall-service ovirt-fence-kdump-listener ovirt-imageio-proxy ovirt-websocket-proxy ovirt-http ovirt-vmconsole-proxy ovirt-provider-ovn https http cockpit libvirt-tls snmp vdsm ovirt-imageio ovirt-vmconsole ports: 9986/tcp 22/tcp 6081/udp 6100/tcp protocols: masquerade: yes forward-ports: port=1337:proto=tcp:toport=22:toaddr=192.168.60.3 port=80:proto=tcp:toport=80:toaddr=192.168.60.5 source-ports: icmp-blocks: rich rules: rule family="ipv4" source address="0.0.0.0/24" accept [root@athena ~]# Portfowarding also works to the host 192.168.60.3. so that is no poiint. But only traffic from the insidehost to outside is not working. Thanks

1 0

Rebrandind Problems
by siovelrm＠gmail.com 19 Apr '19

19 Apr '19

hello I made a rebranding of my ovirt 4.3.2, but something went wrong and I saved it from the original incorrectly without realizing it. Please I need the original "ovirt.brand" and "ovirt" directories when ovirt 4.3.2 is installed. Where can I get them to be able to restore them? Greetings

2 1

Re: Expand existing gluster storage in ovirt 4.2/4.3
by Strahil 18 Apr '19

18 Apr '19

Recently , it was discussed in the mailing lists and dev mentioned that distributed replicated volumes are not officially supported, but some users use them. Even if not supported, it still should work without issues.If you think not to go this way, you can create a new 3 node cluster which will be fully suppported. Otherwise, if you go towards replicated distributed volumes , you just need to provide another set of 3 bricks and once added you can rebalance in order to distribute the files accross the sets. Here is an old thread that describes it for replica 2 volume types: https://lists.gluster.org/pipermail/gluster-users/2011-February/006599.html I guess I have confused you with my last e-mail, but that was not intentionable. Best Regards, Strahil NikolovOn Apr 17, 2019 17:13, adrianquintero(a)gmail.com wrote: > > Hi Strahil, > I had a 3 node Hyperconverged setup and added 3 new nodes to the cluster for a total of 6 servers. I am now taking advantage of more compute power, however the gluster storage part is what gets me. > > Current Hyperconverged setup: > - host1.mydomain.com > Bricks: > engine > data1 > vmstore1 > - host2.mydomain.com > Bricks: > engine > data1 > vmstore1 > - host3.mydomain.com > Bricks: > engine > data1 > vmstore1 > > - host4.mydomain.com > Bricks: > > - host5.mydomain.com > Bricks: > > - host6.mydomain.com > Bricks: > > > As you can see from the above, the original first 3 servers are the only ones that contain the gluster storage bricks, so storage redundancy is not set across all 6 nodes. I think it is a lack of understanding from my end on how ovirt and gluster integrate with one another so have a few questions: > > How would I go about achieving storage redundancy across all nodes? > Do I need to configure gluster volumes manually through the OS CLI? > If I configure the fail storage scenario manually will oVirt know about it? > > Again I know that the bricks must be added in sets of 3 and per the first 3 nodes my gluster setup looks like this (all done by hyperconverged seup in ovirt): > engine volume: host1:brick1, host2:brick1, host3:brick1 > data1 volume: host1:brick2, host2:brick2, host3:brick2 > vmstore1 volume: host1:brick3, host2:brick3, host3:brick3 > > So after adding the 3 new servers I dont know if I need to do something similar to the example in https://medium.com/@tumballi/scale-your-gluster-cluster-1-node-at-a-time-62…, if I do a similar change will oVirt know about it? will it be able to handle it as hyperconverged? > > As I mentioned before I normally see 3 node hyperconverged setup examples with gluster but have not found one for 6, 9 or 12 node cluster. > > Thanks again. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/U5T7TCSP4HFB2…

1 0

Ovirt Host Replacement/Rebuild
by Jim Kusznir 18 Apr '19

18 Apr '19

Hi all: I had an unplanned power outage (generator failed to start, power failure lasted 3 min longer than UPS batteries). One node didn't survive the unplanned power outage. By that, I mean it kernel panic's on boot, and I haven't been able to capture the KP or the first part of it (just the end), and so I don't truely know what the root cause is. I have validated the hardware is just fine, so its got to be an OS corruption. Based on this, I was thinking that perhaps the easiest way to recover would simply be to delete the host from the cluster, reformat and reinstall this host, and then add it back to the cluster as a new host. Is this in fact a good idea? Are there any references to how to do this (the detailed steps so I don't mess it up)? My cluster is (was) a 3 node hyperconverged cluster with gluster used for the management node. I also have a gluster share for VMs, but I use an NFS share from a NAS for that (which I will ask about in another post). Thanks for the help! --Jim

2 1

Importing existing GlusterFS
by Zryty ADHD 18 Apr '19

18 Apr '19

Hi, I have a questiion about that. I Install Ovirt 4.3.3 on Rhel 7.6 and want to import my existing GlusterFs cluster to it but i can't find option to do that. Can anyone explain me how to do that or its not possible in this version ?

3 5

oVirt and NetApp NFS storage
by klaasdemter＠gmail.com 18 Apr '19

18 Apr '19

Hi, I got a question regarding oVirt and the support of NetApp NFS storage. We have a MetroCluster for our virtual machine disks but a HA-Failover of that (active IP gets assigned to another node) seems to produce outages too long for sanlock to handle - that affects all VMs that have storage leases. NetApp says a "worst case" takeover time is 120 seconds. That would mean sanlock has already killed all VMs. Is anyone familiar with how we could setup oVirt to allow such storage outages? Do I need to use another type of storage for my oVirt VMs because that NFS implementation is unsuitable for oVirt? Greetings Klaas

2 4

what is the best solution for gluster?
by Edoardo Mazza 18 Apr '19

18 Apr '19

Hi all, I have 4 nodes with ovirt and gluster and I must create a new gluster volume, but I would know which is the best solution to have a high avaibility, best performance without wasting much space in the disk. Thanks for suggestions Edoardo

1 0

Changing from thin provisioned to preallocated?
by Wesley Stewart 18 Apr '19

18 Apr '19

I am currently running a ZFS server (Running RaidZ2) and I have been experimenting with NFS and shares to host my guests. I am currently running oVirt 4.2.8 and using a RaidZ2 NFS mount for the guests. ZFS definitely is definitely pretty awful (At least in my experience so far) for hosting VMs. I believe this is due to the synchronous writes being performed. However, I think running an iSCSI target with Synchronization disabled over a 10Gb connection might do the trick. (I have a couple of mirroed SSD drives for performance if I need it, but the RaidZ2 crawls for disk speed). When I tried to migrate a thin provisioned guest to iSCSI, I keep getting an "Out of disk space error" which I am pretty sure is due to the block style storage on the iSCSI target. Is there a way to switch from Thin to Preallocated? Or is my best bet to try and take a snapshot and clone this into a pre-allocated disk?

3 3

[ANN] ovirt 4.3.3 async update is now available
by Sandro Bonazzola 17 Apr '19

17 Apr '19

The oVirt Team has just released a new version of the following packages: - cockpit-ovirt-0.12.8 - ovirt-release43-4.3.3.1 - oVirt Node The async release addresses the following bugs: Bug 1697965 <https://bugzilla.redhat.com/show_bug.cgi?id=1697965> - [gluster-ansible] Order of the hosts are not preserved, when creating the volume This fix is relevant only on new deployments. -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

Re: Expand existing gluster storage in ovirt 4.2/4.3
by Strahil 17 Apr '19

17 Apr '19

You can have hyperconverged setup, but the number of bricks should be multiplied by 3. Best Regards, Strahil NikolovOn Apr 11, 2019 14:52, adrianquintero(a)gmail.com wrote: > > Thank you for the explanation, so from my understanding we can't have a hyperconverged setup if more that 3 nodes are required? or is it possible if gluster is setup separate of the oVirt node installs? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/77URQGHY7LBF4…

2 1

Problem trying to detach ISO datastore
by Eduardo Mayoral 17 Apr '19

17 Apr '19

Hi. When trying to detach an ISO datastore I get the following error. Also fails if I try to switch the datastore from maintenance to active. 2019-04-17 10:07:23,100+0000 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.01 seconds (__init__:312) 2019-04-17 10:07:23,104+0000 INFO (jsonrpc/4) [vdsm.api] START detachStorageDomain(sdUUID=u'2722942e-0887-439c-81f0-70b60d860060', spUUID=u'6901a281-c414-42c6-8b41-3cf6e6bcd788', msdUUID=u'00000000-0000-0000-0000-000000000000', masterVersion=14, options=None) from=::fff f:10.5.72.63,58618, flow_id=0420c8d0-fb37-4b22-9fea-9ee8bac7d41d, task_id=9301405a-eaf6-4f1c-a1b6-6537740e2daf (api:48) 2019-04-17 10:07:23,105+0000 INFO (jsonrpc/4) [storage.StoragePool] sdUUID=2722942e-0887-439c-81f0-70b60d860060 spUUID=6901a281-c414-42c6-8b41-3cf6e6bcd788 (sp:1048) 2019-04-17 10:07:23,316+0000 INFO (itmap/0) [IOProcessClient] (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt3) Starting client (__init__:308) 2019-04-17 10:07:23,324+0000 INFO (itmap/1) [IOProcessClient] (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt) Starting client (__init__:308) 2019-04-17 10:07:23,327+0000 INFO (ioprocess/74639) [IOProcess] (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt3) Starting ioprocess (__init__:434) 2019-04-17 10:07:23,333+0000 INFO (ioprocess/74645) [IOProcess] (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt) Starting ioprocess (__init__:434) 2019-04-17 10:07:23,335+0000 INFO (jsonrpc/4) [IOProcessClient] (2722942e-0887-439c-81f0-70b60d860060) Starting client (__init__:308) 2019-04-17 10:07:23,344+0000 INFO (ioprocess/74653) [IOProcess] (2722942e-0887-439c-81f0-70b60d860060) Starting ioprocess (__init__:434) 2019-04-17 10:07:23,350+0000 INFO (jsonrpc/4) [storage.StorageDomain] Removing remnants of deleted images [] (fileSD:740) 2019-04-17 10:07:23,352+0000 INFO (jsonrpc/4) [vdsm.api] FINISH detachStorageDomain error=Storage domain not in pool: u'domain=2722942e-0887-439c-81f0-70b60d860060, pool=6901a281-c414-42c6-8b41-3cf6e6bcd788' from=::ffff:10.5.72.63,58618, flow_id=0420c8d0-fb37-4b22-9fea- 9ee8bac7d41d, task_id=9301405a-eaf6-4f1c-a1b6-6537740e2daf (api:52) 2019-04-17 10:07:23,352+0000 ERROR (jsonrpc/4) [storage.TaskManager.Task] (Task='9301405a-eaf6-4f1c-a1b6-6537740e2daf') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in detachStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 891, in detachStorageDomain pool.detachSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1053, in detachSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 551, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=2722942e-0887-439c-81f0-70b60d860060, pool=6901a281-c414-42c6-8b41-3cf6e6bcd788' 2019-04-17 10:07:23,352+0000 INFO (jsonrpc/4) [storage.TaskManager.Task] (Task='9301405a-eaf6-4f1c-a1b6-6537740e2daf') aborting: Task is aborted: "Storage domain not in pool: u'domain=2722942e-0887-439c-81f0-70b60d860060, pool=6901a281-c414-42c6-8b41-3cf6e6bcd788'" - co de 353 (task:1181) 2019-04-17 10:07:23,353+0000 ERROR (jsonrpc/4) [storage.Dispatcher] FINISH detachStorageDomain error=Storage domain not in pool: u'domain=2722942e-0887-439c-81f0-70b60d860060, pool=6901a281-c414-42c6-8b41-3cf6e6bcd788' (dispatcher:83) 2019-04-17 10:07:23,353+0000 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StorageDomain.detach failed (error 353) in 0.25 seconds (__init__:312) 2019-04-17 10:07:26,275+0000 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:10.5.72.63,58604 (api:48) 2019-04-17 10:07:26,275+0000 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:10.5.72.63,58604 (api:54) 2019-04-17 10:07:26,275+0000 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312) 2019-04-17 10:07:29,675+0000 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:10.5.72.63,58604 (api:48) 2019-04-17 10:07:29,739+0000 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:10.5.72.63,58604, task_id=3ec94f9f-7c98-4c49-8710-ffbfd9910ff2 (api:48) 2019-04-17 10:07:29,739+0000 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'a97bbda5-33e4-4e78-890c-f7d2a572179e': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000505524', 'lastCheck': '3.6', 'valid': True}, u'4121f681-45c3-413a-9c6e-6 9d154710ace': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000790301', 'lastCheck': '2.7', 'valid': True}} from=::ffff:10.5.72.63,58604, task_id=3ec94f9f-7c98-4c49-8710-ffbfd9910ff2 (api:54) 2019-04-17 10:07:29,740+0000 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:10.5.72.63,58604, task_id=f567e03a-8cf4-49dd-84eb-628acb10e1d3 (api:48) 2019-04-17 10:07:29,740+0000 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.5.72.63,58604, task_id=f567e03a-8cf4-49dd-84eb-628acb10e1d3 (api:54) Interestingly, I have 2 datacenters, I can attach, detach, set to maintenance or activate this same datastore on the second datacenter, but not on this one. So far I have tried updating OVF, setting master datastore and hosts to maintenance and re-activating them. Started happening on 4.3.2, updating to 4.3.3 makes no difference. This is what I see on the database. Looks OK to me: engine=# select * from storage_pool; id | name | description | storage_pool_type | storage_pool_format_type | status | master_domain_version | spm_vds_id | compatibility_version | _create_date | _update_date | quota_enforcement_type | free_text_comment | is_local --------------------------------------+-------------------+-------------+-------------------+--------------------------+--------+-----------------------+--------------------------------------+-----------------------+-------------------------------+----------------------- --------+------------------------+-------------------+---------- 039aef77-77c8-4f04-8d17-22f4b655df28 | dc-pre1-vc03-cl02 | | | 4 | 1 | 1 | 42c969f3-a2a6-4784-8506-e6fbc731da9d | 4.3 | 2019-03-14 12:13:24.411468+00 | 2019-04-17 08:38:05.53 8042+00 | 0 | | f 6901a281-c414-42c6-8b41-3cf6e6bcd788 | dc-pre1-vc03-cl01 | | | 4 | 1 | 15 | 089cdde4-ecbf-4054-82f9-c106b046d73a | 4.3 | 2019-03-14 12:13:24.365529+00 | 2019-04-17 11:10:46.42 2305+00 | 0 | | f (2 rows) engine=# select * from storage_pool_iso_map ; storage_id | storage_pool_id | status --------------------------------------+--------------------------------------+-------- 2722942e-0887-439c-81f0-70b60d860060 | 039aef77-77c8-4f04-8d17-22f4b655df28 | 6 2722942e-0887-439c-81f0-70b60d860060 | 6901a281-c414-42c6-8b41-3cf6e6bcd788 | 6 5d390949-24ec-45e2-bb6d-43b3f30bd286 | 039aef77-77c8-4f04-8d17-22f4b655df28 | 3 94cb1a4f-a267-479c-88e7-ea9c32e3fd24 | 039aef77-77c8-4f04-8d17-22f4b655df28 | 3 4121f681-45c3-413a-9c6e-69d154710ace | 6901a281-c414-42c6-8b41-3cf6e6bcd788 | 3 a97bbda5-33e4-4e78-890c-f7d2a572179e | 6901a281-c414-42c6-8b41-3cf6e6bcd788 | 3 (6 rows) Apart from root cause. I would be interested in recovering from the error, detaching the ISO storage, even if it is manually. This datacenter is not in production yet, so I have options on the things I can try. -- Eduardo Mayoral Jimeno Systems engineer, platform department. Arsys Internet. emayoral(a)arsys.es - +34 941 620 105 - ext 2153

1 1

Re: restore from dump
by Yedidyah Bar David 17 Apr '19

17 Apr '19

On Tue, Apr 16, 2019 at 10:10 PM Florian Rädler <Florian.Raedler(a)deutschebahn.com> wrote: > > Hi, > > > > is it possible to restore an engine from dump file? Define "restore an engine" and "dump file". Assuming "restore an engine" means "Install a fresh OS+engine and make it behave like an old one", and "dump file" being a pg_dump dump file, the answer is "not easily, and not fully". You will have to do something like: 1. Install OS+engine 2. Manually provision PG (init, create user/db, etc) 3. Restore your dump (pg_restore or psql, depending on format). Use the user you created, not postgres, for restoring - this will have much better chances to get permissions right 4. Run engine-setup and provide it the credentials of the database you restored It should mostly work, and I heard about people doing that, although didn't try this myself. Notably, you will have to reinstall all your hosts (move to maintenance, reinstall). If you have also a backup of /etc, that's a different case. Then, you can selectively restore files there before engine-setup. To see a list of the files you should restore, check the source of the engine-backup script - it has an internal variable called 'BACKUP_PATHS'. Good luck and best regards, -- Didi

2 1

VM migrations failing after upgrade 4.3.2 -> 4.3.3
by Eduardo Mayoral 17 Apr '19

17 Apr '19

Hi, After upgrade to 4.3.3 from 4.3.2, VM migrations are failing with "No available host to migrate VMs to" Interestingly, this happens in one of our clusters, not on the other. Both on the same datacenter, both in 4.3 compatibility version. This are the relevant lines from /var/log/ovirt-engine/engine.log 2019-04-17 08:58:41,816Z ERROR [org.ovirt.engine.core.bll.GetValidHostsForVmsQuery] (default task-7) [51191ffb-0ae1-4414-8916-9e5012d86289] Query 'GetValidHostsForVmsQuery' failed: null 2019-04-17 08:58:41,816Z ERROR [org.ovirt.engine.core.bll.GetValidHostsForVmsQuery] (default task-7) [51191ffb-0ae1-4414-8916-9e5012d86289] Exception: java.lang.NullPointerException at org.ovirt.engine.core.bll.scheduling.SchedulingManager.subtractRunningVmResources(SchedulingManager.java:923) [bll.jar:] at org.ovirt.engine.core.bll.scheduling.SchedulingManager.canSchedule(SchedulingManager.java:616) [bll.jar:] at org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.lambda$getValidHosts$0(GetValidHostsForVmsQuery.java:56) [bll.jar:] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) [rt.jar:1.8.0_201] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) [rt.jar:1.8.0_201] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) [rt.jar:1.8.0_201] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [rt.jar:1.8.0_201] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) [rt.jar:1.8.0_201] at org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.getValidHosts(GetValidHostsForVmsQuery.java:59) [bll.jar:] at org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.executeQueryCommand(GetValidHostsForVmsQuery.java:36) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:521) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:490) [bll.jar:] at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) [:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view2.runQuery(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.runQuery(BackendResource.java:82) at org.ovirt.engine.api.restapi.resource.BackendResource.getBackendCollection(BackendResource.java:152) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:68) at org.ovirt.engine.api.restapi.resource.BackendHostsResource.listAll(BackendHostsResource.java:83) at org.ovirt.engine.api.restapi.resource.BackendHostsResource.list(BackendHostsResource.java:54) at org.ovirt.engine.api.resource.HostsResource.doList(HostsResource.java:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] 2019-04-17 08:58:41,831Z ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-7) [] Operation Failed: 2019-04-17 08:58:41,831Z ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-7) [] Exception: org.ovirt.engine.api.restapi.resource.BaseBackendResource$BackendFailureException: at org.ovirt.engine.api.restapi.resource.BackendResource.backendFailure(BackendResource.java:412) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.getBackendCollection(BackendResource.java:155) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:68) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendHostsResource.listAll(BackendHostsResource.java:83) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendHostsResource.list(BackendHostsResource.java:54) [restapi-jaxrs.jar:] at org.ovirt.engine.api.resource.HostsResource.doList(HostsResource.java:130) [restapi-definition.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.0.15.Final.jar:2.0.15.Final] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] -- Eduardo Mayoral Jimeno Systems engineer, platform department. Arsys Internet. emayoral(a)arsys.es - +34 941 620 105 - ext 2153

2 2

Inconsistencies beteween WebUI, API read verbs and API write verbs
by Baptiste Agasse 17 Apr '19

17 Apr '19

Hi All, As I stated in a previous thread, we use instances types and wanted to automate some parts to manage it [1]. Thansk to ovirt-orb/lago and pointers given in the last thread, I'm working on ansible an module to manage ovirt instance types, but I'm facing some troubles for some of instance type attributes. From the API, I don't have access to all fields provided via WebUI (eg: Custom Emulated Machine), and it seems that some fields can be added/updated via the API but are not part of the response when you read an instance type (eg: VirtIO-SCSI Enabled). Provided examples are not exhaustive. I tried to find my way in the oVirt engine code-base to see if the fix will be easy for a non java developer like me. I find, I think, that it seems to be some API definitions for instance type [2] that define, if I understand it well, all verbs that can be applied to each resource and what field can be put when adding an instance type, but nothing about fields to be included when you read a resource (and I'm not sure if its related to v4 API). I find also some java code [3] [4] that seems related to this but As I'm not a java developer, its a little bit difficult for me to find my path in it. For me this inconsistencies are a bug, but as you stated that instance types are in deep maintenance mode, do you think that bug will get some attention ? My objective is to have a ansible module that manage the hardware/bios related stuff (memory, boot sequence, scsi controller, bios type, almost all the things) and HA for instance type. Have a nice day. Cheers. [1]: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TJAXYBMSIAAAY… [2]: https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/r… [3]: https://github.com/oVirt/ovirt-engine/blob/ede62008318d924556bc9dfc5710d90e… [4]: https://github.com/oVirt/ovirt-engine/blob/ede62008318d924556bc9dfc5710d90e… [5]: https://ovirt.github.io/ovirt-engine-sdk/4.1/types.m.html#ovirtsdk4.types.I… -- Baptiste

2 2

oVirt nVidia vGPU driver questions
by femi adegoke 17 Apr '19

17 Apr '19

I'm on oVirt 4.2.6. 1) If I have a Tesla P4 (Pascal) GPU (non-GRID), do I use nVidia vGPU driver for RHEL KVM 6.2 or the Linux KVM vGPU 6.2 driver? 2) If I have a Grid K1 or K2, I assume the correct driver would be GRID 4.7? 3) Must the vGPU driver be installed on the host & in the vm? 4) Must the vGPU driver be installed first before I can run "yum -y install vdsm-hook-vfio-mdev”

5 6

oVirt 4.3 IPv6 only networking - some basic flows demonstration
by Eitan Raviv 17 Apr '19

17 Apr '19

Hi, I posted a video [1] with a short demo about flows that since 4.3 support IPv6 only networking. Enjoy... Thanks [1] https://youtu.be/h45dY4LA0CI

1 0

restore from dump
by Florian Rädler 16 Apr '19

16 Apr '19

Hi, is it possible to restore an engine from dump file? Thanks for your help. ________________________________ Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20190408> Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz

1 0

Cant install oVirt hosted engine 4.3
by dskryminskiy＠devexperts.com 16 Apr '19

16 Apr '19

I do a fresh install oVirt 4.3 on CentOS 7.6.1810, but in the end of playbook it falls down: [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Force facts gathering] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] ok: [localhost] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190415123351.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20190415122114-rk9ve5.log Log files can be found here: https://ufile.io/kylza1g2

3 9