- Users - oVirt List Archives

[ANN] oVirt 4.3.8 First Release Candidate is now available for testing
by Sandro Bonazzola 06 Dec '19

06 Dec '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.8 First Release Candidate for testing, as of December 6th, 2019. This update is a release candidate of the eighth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available Additional Resources: * Read more about the oVirt 4.3.8 release highlights: http://www.ovirt.org/release/4.3.8/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.8/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Gluster FS & hosted engine fails to set up
by rob.downer＠orbitalsystems.co.uk 05 Dec '19

05 Dec '19

I have set up 3 new servers and as you can see Gluster is working well, however the hosted engine deployment fails can anyone suggest a reason ? I have wiped and set up all three servers again and set up Gluster first. This is the gluster congig I have used for the setup. Please review the configuration. Once you click the 'Finish Deployment' button, the management VM will be transferred to the configured storage and the configuration of your hosted engine cluster will be finalized. You will be able to use your hosted engine once this step finishes. * StorageStorage Type:glusterfs Storage Domain Connection:gfs3.gluster.private:/engine Mount Options:backup-volfile-servers=gfs2.gluster.private:gfs1.gluster.private Disk Size (GiB):58 [ INFO ] TASK [ovirt.hosted_engine_setup : Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Unexpected exception]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Unexpected exception]\". HTTP response code is 400."} root@ovirt3 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs3.gluster.private:/gluster_bricks/ data/data 49152 0 Y 3756 Brick gfs2.gluster.private:/gluster_bricks/ data/data 49153 0 Y 3181 Brick gfs1.gluster.private:/gluster_bricks/ data/data 49152 0 Y 15548 Self-heal Daemon on localhost N/A N/A Y 17602 Self-heal Daemon on gfs1.gluster.private N/A N/A Y 15706 Self-heal Daemon on gfs2.gluster.private N/A N/A Y 3348 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs3.gluster.private:/gluster_bricks/ engine/engine 49153 0 Y 3769 Brick gfs2.gluster.private:/gluster_bricks/ engine/engine 49154 0 Y 3194 Brick gfs1.gluster.private:/gluster_bricks/ engine/engine 49153 0 Y 15559 Self-heal Daemon on localhost N/A N/A Y 17602 Self-heal Daemon on gfs1.gluster.private N/A N/A Y 15706 Self-heal Daemon on gfs2.gluster.private N/A N/A Y 3348 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs3.gluster.private:/gluster_bricks/ vmstore/vmstore 49154 0 Y 3786 Brick gfs2.gluster.private:/gluster_bricks/ vmstore/vmstore 49152 0 Y 2901 Brick gfs1.gluster.private:/gluster_bricks/ vmstore/vmstore 49154 0 Y 15568 Self-heal Daemon on localhost N/A N/A Y 17602 Self-heal Daemon on gfs1.gluster.private N/A N/A Y 15706 Self-heal Daemon on gfs2.gluster.private N/A N/A Y 3348 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks

1 0

Re: Importing a Virtual Machine from a KVM Host
by Dirk Streubel 05 Dec '19

05 Dec '19

Hello Paul, thanks a lot for your answer. If i have any other questions during the import maybe i can asked again? And i let you know the result :) Regards Dirk Am 05.12.19 um 13:22 schrieb Staniforth, Paul: > Hello Dirk, > The link you are using is old developer docs, try the following. > https://www.ovirt.org/documentation/admin-guide/chap-External_Providers.html > > You need to setup an external provider and use a proxy host in the datacentre (not the engine host) you want to import into, this needs to be on the same network as your KVM host and have the virt-v2v package installed. > > Regards, > Paul S. > > -----Original Message----- > From: Dirk Streubel <dirk.streubel(a)posteo.de> > Sent: 05 December 2019 11:10 > To: users <users(a)ovirt.org> > Subject: [ovirt-users] Importing a Virtual Machine from a KVM Host > > Hello everbody, > > i am just a little bit confused about importing Qemu Images from my KVM Host. > > At home, and at work i am using a ovirt engine in a VM. The Version is 4.3.7 and on bare metal my host. > > Everything works fine but i have the same problem at home and on work. I want to import my qcow2 Images from a KVM Host but i > > will not work. > > The command virsh -r -c 'qemu+tcp://root@host1.example.org/system' list --all shows me akk my VMs. > > But when i want to load the images in the GUI, nothing happens. I follow this instruction: > > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > > After that, i take a look on the official 4.3 Rhev Documentation from Red Hat. There stand this: > > Log in to the proxy host and generate SSH keys for the vdsm user. > > sudo -u vdsm ssh-keygen > > So, i have no proxy host. So i want to do this on my ovirt-engine. Is this right? > > Next thing: the user vdsm is locked on the ovirt-engine machine and the host. > > So, do i have to unlocked the account of the user vdsm or what is the best way. > > My Problem is i found nothing that would help me to solve this problem. > > Maybe somebody here do me a favour and help me to solve this "challenge" > > Regards > > Dirk > > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/

1 0

Possible sources of cpu steal and countermeasures
by klaasdemter＠gmail.com 05 Dec '19

05 Dec '19

Hi, I'm having performance issues with a ovirt installation. It is showing high steal (5-10%) for a cpu intensive VM. The hypervisor however has more than 65% of his resources idle while the steal is seen inside of the VM. Even when placing only a single VM on a hypervisor it still receives steal (0-2%), even though the hypervisor is not overcommited. Hypervisor: 2 Socket system in total 2*28(56HT) cores VM: 30vCPUs (ovirt seems to think its a good idea to make that 15 sockets * 2 cores) My questions are: a) Could it be that the hypervisor is trying to schedule all 30 cores on a single numa node, ie using the HT cores instead of "real" ones and this shows up as steal? b) Do I need to make VMs this big numa-aware and spread the vm over both numa nodes? c) Would using the High Performance VM type help in this kind of situation? d) General advise: how do I reduce steal in an environment where the hypervisor has idle resources Any advise would be appreciated. Greetings Klaas

3 2

Re: Unable to attach ISO domain to Datacenter
by Strahil 05 Dec '19

05 Dec '19

Hi Ivan, Which file is this ? Contact me in private and I will send you my copy for comparison. Best Regards, Strahil NikolovOn Dec 4, 2019 18:55, Ivan Apolonio <ivan(a)apolonio.com.br> wrote: > > I've tried to commenting out this line on /etc/pam.d/password-auth: > > auth requisite pam_succeed_if.so uid >= 1000 quiet_success > > However, although it has stopped complaining about "requirement uid >= 1000 not met by user vdsm" on /var/log/secure , the error trying to attach ISO Domain to Datacenter still persists, showing the same error message, so it looks that this was not the cause of the problem :-( > > Going back to stage zero. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TTD4PH2FIUK23…

1 0

Ovirt-OSN integration
by siovelrm＠gmail.com 04 Dec '19

04 Dec '19

Hi, I want to use External Logical Networks Openstack Neutron. For this it is necessary to have all Openstack installed or is it possible to have only Neutron and used as a service? Sorry if the question is very obvious but I am starting now with this topic.

2 2

Re: 3node HCI fails when HostedEngineLocal is trying to add additional Gluster members
by Strahil 04 Dec '19

04 Dec '19

If you manage to get your VM on the gluster - you are almost done. I had similar situation and using virsh/hosted-engine , I have managed to reach the GUI. From there we can have a Clue what is going on. Usually there is a dependency: - Master storage domain should be UP -This allows the DC to become UP -Then a host in the cluster can become SPM (control storage) -And only then you can start your other gluster volumes. So I can only recommend you to try to reach GUI and check the situation. Then it will be moderately easy to bring everything up. Best Regards, Strahil NikolovOn Dec 4, 2019 17:06, thomas(a)hoberg.net wrote: > > Thanks Strahil, for your suggestions. > > Actually, I was far beyond the pick-up point you describe, as the Gluster had all been prepared and was operable, even the local VM was already running and accessible via the GUI. > > But I picked up your hint to try to continue with the scripted variant, and found that it allowed me much better insight into what was going on. > > I am a little worried, though, that it actually works a little differently from the GUI-wizard variant, in any case the failures don't seem identical. > That would have implications in terms of test-automation, which I'd rather not have to worry about. > > In a separate test for the same operation on a separate set of hardware the installation got significantly further, up to the point where the local VM had actually been moved onto the Gluster storage and into the Cluster, but then failed a validation step at the very end (while the VM is actually up and running, albeit only withe primary host and that listed as "Unresponsive" while it's hosting the VM....) > > I have opened a separate tickt for that... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FTCFIP7Q5KWXT…

1 0

Cannot activate/deactivate storage domain
by Albl, Oliver 04 Dec '19

04 Dec '19

Hi all, I run an oVirt 4.3.6.7-1.el7 installation (50+ hosts, 40+ FC storage domains on two all-flash arrays) and experienced a problem accessing single storage domains. As a result, hosts were taken "not operational" because they could not see all storage domains, SPM started to move around the hosts. oVirt messages start with: 2019-11-04 15:10:22.739+01 | VDSM HOST082 command SpmStatusVDS failed: (-202, 'Sanlock resource read failure', 'IO timeout') 2019-11-04 15:10:22.781+01 | Invalid status on Data Center <name>. Setting Data Center status to Non Responsive (On host HOST82, Error: General Exception). ... 2019-11-04 15:13:58.836+01 | Host HOST017 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.85+01 | Host HOST005 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.85+01 | Host HOST012 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.851+01 | Host HOST002 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.851+01 | Host HOST010 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.851+01 | Host HOST011 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.852+01 | Host HOST004 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.011+01 | Host HOST017 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.238+01 | Host HOST004 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.249+01 | Host HOST005 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.255+01 | Host HOST012 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.273+01 | Host HOST002 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.279+01 | Host HOST010 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.386+01 | Host HOST011 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:15:14.145+01 | Storage domain HOST_LUN_221 experienced a high latency of 9.60953 seconds from host HOST038. This may cause performance and functional issues. Please consult your Storage Administrator. The problem mainly affected two storage domains (on the same array) but I also saw single messages for other storage domains (one the other array as well). Storage domains stayed available to the hosts, all VMs continued to run. When constantly reading from the storage domains (/bin/dd iflag=direct if=<metadata> bs=4096 count=1 of=/dev/null) we got expected 20+ MBytes/s on all but some storage domains. One of them showed "transfer rates" around 200 Bytes/s, but went up to normal performance from time to time. Transfer rate to this domain was different between the hosts. /var/log/messages contain qla2xxx abort messages on almost all hosts. There are no errors on SAN switches or storage array (but vendor is still investigating). I did not see high load on the storage array. The system seemed to stabilize when I stopped all VMs on the affected storage domain and this storage domain became "inactive". Currently, this storage domain still is inactive and we cannot place it in maintenance mode ("Failed to deactivate Storage Domain") nor activate it. OVF Metadata seems to be corrupt as well (failed to update OVF disks <id>, OVF data isn't updated on those OVF stores). The first six 512 byte blocks of /dev/<id>/metadata seem to contain only zeros. Any advice on how to proceed here? Is there a way to recover this storage domain? All the best, Oliver

4 13

3node HCI fails when HostedEngineLocal is trying to add additional Gluster members
by thomas＠hoberg.net 04 Dec '19

04 Dec '19

Three Gen8 HP360 recalled from retirement with single 1TB TLC SATA SSD for boot and oVirt /engine and 7x4TB HDD RAID6 for /vmstore and /data, 10Gbit NICs and network. All CentOS 7.7 updated daily. These machines may not be used exclusively for oVirt so I don't want to re-install the OS, when an oVirt setup fails: Instead I try my best to clean up the nodes when doing another oVirt installation run. They ran oVirt for a week or two using a completely distinct set of storage, so they are fundamentally sound, but we wanted higher storage capacity so I swapped everything and re-installed CentOS very much the same way as before. The first oVirt setup went smoothly but the cluster crumbled without much usage. I won't go into details here, because I didn't want to investigate for now, instead I focussed on redoing the installation and cleaning up the old setup. I know the docs actually recommend starting with wiped hardware, but operationally that would be a show-stopper for the intended use case. So I cleaned up the best I can (ovirt-hosted-engine-cleanup with and without redoing the whole Gluster storage setup, where apart from SSD caching not working, I don't have issues). Undoing the network changes in such a way that the oVirt HCI wizard ceases complaining is a bit more involved. I typically run: - vdsm-tool ovn-unconfigure - vdsm-tool clear-nets (now need to switch to the console) - vdsm-tool remove-config and then I still need to edit /etc/sysconfig/network-scripts/ifcfg-<ethernet-device> to bring the physical adapter back to life. Sometimes I still need to remove the ovirtmgmnt bridge manually etc. Whether I remove and redo the Gluster as a bit of an effect in re-installation, but it doesn't make a difference in what follows. So here is where I am currently getting stuck consistently: The wizard is gone through preparing the Gluster storage (which is completely functional at that point), has created the local VM on the installation node, installed the Postgres database, filled it etc. basically has oVirt up and running with the primary Gluster node and now would like to add the second and third nodes. At that point I get "Connection lost" in the Web-Wizard, evidently as a consequence of Ansible fiddling around heavily to set up the local bridge for the VM. I remember that for the scripted variant of the setup it is recommended to run the script behind 'screen' or 'tmux' in order to ensure its execution isn't interrupted by that. But for the GUI variant, evidently there *should* be some other type of potection, perhaps via the re-connecting nature of HTTP... Pushing the "Reconnect" button on the GUI at that point doesn't return you to the point of the setup, but only offers to redeploy, while the HostedEngineLocal is still there and running. I ssh'd into the machine and started looking for errors and warning and saw that the installation had gone rather far without incidence. OTOPI had completely finished the WildFly server is up and running the Postgres database fully installed and running smoothly, the only thing I can find is that it's trying to add the additional gluster nodes, but complains that these nodes (quotes gluster-UUIDs) are not part of the "cluster". An investigation into the Postgres database shows, that the 'gluster_server' table indeed only has the primary node in it. I don't know what part of the process should have added the other two nodes, but there seems to be no *remaining* connectivity issue with the Gluster members. I installed gscli and connected to all three nodes and volumes without issue. I am guessing at this point, that the complex rewiring of the software defined network is causing a temporary issue and a race condition that I don't know how to recover from. Since the oVirt management GUI is actually fully operational and can be reached from the primary node via the temporary bridge, I went into the GUI and even managed to add the additional two nodes without any problems. Their installation went through without any issues, they showed up in the gluster_servers table on Postgress and basically the installation could have proceeded from that point, except... that I don't know how to restart the process from that point: It still has to 'beam' the local VM into the Gluster storage and restart it there. I have gone through the process three times now, with absolutely identical results. I could use some help how to recover from that situation, which looks like a race condition, nothing a re-installation of everything would really resolve. In the mean-time, I'll try the scripted variant on 'screen' to see if that fares better.

2 3

Re: Certificate of host is invalid
by Strahil 04 Dec '19

04 Dec '19

Hi , You can try with: 1. Set the host in maintenance 2. From Install dropdown , select 'reinstall' and then configure the necessary info + whether you would like to use the host as Host for the HostedEngine VM. Once the reinstall (of Ovirt software) is OK, the node will be activated automatically. Best Regards, Strahil NikolovOn Nov 27, 2019 18:01, Jon bae <jonbae77(a)gmail.com> wrote: > > Hello everybody, > since last update to 4.3.7 I get this error message: > > Certificate of host host.name is invalid. The certificate doesn't contain valid subject alternative name, please enroll new certificate for the host. > > Have you an idea of how I can fix that? > > Regards > Jonathan

3 3

Re: Scenario: Ovirt-engine on hardware and recovery
by Yedidyah Bar David 04 Dec '19

04 Dec '19

On Tue, Dec 3, 2019 at 3:44 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > > Didi, > > Thanks for the reply. > > Yes, backups are done, I am just thinking in the case of Murphy's Law where nothing works and the engine is gone. I am also new to oVirt and am always loking for the "proper" backup method as you have laready mentioned using engine-backup. > > Storage is NFS backed and backups are done in case of storage server failure. OK. I suggest, then, that you methodically plan, document and test your environment on various different failure modes. Since you are new to oVirt, it's obvious you still do not have the full mental image of all the components, their behaviour, etc., and even more so how they might react, or be handled, in emergency situations. In particular, spend some time learning about the SPM, master storage domain, HA VMs and how they might be affected by/affecting restore flows, etc. Good luck! > > What is the preferred VM backup method so both disk and VM config is captured? Would rather not have to put agents on all the VM for OS level backups and then have to manually recreate all of the machines. Do all the VM configs get stored on the shared storage just like one of the other virtualization vendors? oVirt does not supply, nor endorse, a specific "preferred method" for this. Rather, it exposes an API, and provides a simple example script: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/vm_backu… There are several backup products/projects, both large and small, both open and closed source, that use this API and can be used. I personally never used any, so can't recommend anything. If you search a bit the list archives you'll find at least some of them. Best regards, > > Appreciate the time from folks on the list who respond. > ________________________________ > From: Yedidyah Bar David <didi(a)redhat.com> > Sent: Tuesday, December 3, 2019 3:29 AM > To: Robert Webb <rwebb(a)ropeguru.com> > Cc: users <users(a)ovirt.org> > Subject: Re: [ovirt-users] Scenario: Ovirt-engine on hardware and recovery > > On Tue, Dec 3, 2019 at 4:38 AM <rwebb(a)ropeguru.com> wrote: > > > > So thinking about my setup, I am thinking through different failure scenarios. > > > > So lets say I have a small physical server with 8 cores and 16GB RAM and I install Centos 7 and ovirt-engine on bare metal. This would also be the same scenario is the engine were on a VM. > > > > I run into a major hardware issue and completely lose the engine. > > You do take backups, right? :-) > > For the engine itself, you can use engine-backup. > > > > > How does one recover the cluster setup and not have to start from scratch by having to rebuild all the nodes? Can the engine just be rebuilt and the ovirt nodes be imported? > > You didn't mention your plans re storage. > > If you put your VMs on well-handled storage, and lose everything but > the storage, you can install a new engine, new nodes, and then import > the VMs from the existing storage. You will still miss some > configuration etc., but should mostly be ok. > > If you plan to use gluster storage using the same nodes, aka HCI, the > same applies, obviously, only that the nodes are your storage, and not > an external system. You still have to make sure you handle them > properly - backup, DR, etc., depending on your needs and budget. > > If you do have engine-backup backups, but lost all your storage, > engine-backup will not help you. > > > > > > This scenario is based on the nodes being built from the ovirt node iso. > > That's ok, but is not relevant. Main question is the storage. Can be > external or local/gluster with either ovirt-node or EL. > > Good luck, > > -- > Didi > -- Didi

1 0

Re: Failed export to OVA
by Strahil 04 Dec '19

04 Dec '19

Have you checked /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20191203112412-kvmclst1a.home.whootis.com-67d3d8f2.log ? Best Regards, Strahil NikolovOn Dec 3, 2019 21:36, Geoff Sweet <geoffrsweet+ovirtusers(a)gmail.com> wrote: > > I'm currently running a stand alone server on 4.2.0.2-1.el7.centos. I know this is an older version and I am trying to do some exports for migrations off of it. The Export to OVA seems simple enough but it fails consistently and I really can't figure out why. The logfiles don't provide a lot of information that speaks to me. I double checked the known bug (of older versions) to make sure there is a description set for the disks associated with the VM. I also tried to look in the Ansible logs referenced below but no such log exists. I've also tried multiple different paths for the output. All with the same failure. Below you can see I am attempting to export to /tmp/. Here is the output of the engine logs: > > ###### engine.log ###### > > 2019-12-03 11:15:43,320-08 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3785977) [d25a398d-f650-4911-ae2a-2ca2b54e8396] EVENT_ID: IMPORTEXPORT_STARTING_EXPORT_VM_TO_OVA(1,223), Starting to export Vm GRAYLOG as a Virtual Appliance > 2019-12-03 11:15:43,916-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:15:43,923-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:15:44,261-08 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1 and 5 tasks are waiting in the queue. > 2019-12-03 11:15:44,261-08 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 9 threads waiting for tasks and 0 tasks in queue. > 2019-12-03 11:15:44,261-08 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100 and 100 tasks are waiting in the queue. > 2019-12-03 11:15:44,261-08 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1 and 0 tasks are waiting in the queue. > 2019-12-03 11:15:44,261-08 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5 and 5 tasks are waiting in the queue. > 2019-12-03 11:15:45,979-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:15:45,988-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:15:46,882-08 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now > 2019-12-03 11:15:46,893-08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [] SPMAsyncTask::PollTask: Polling task 'fe1a173f-869c-4522-9984-9c57bace7c87' (Parent Command 'CopyImageGroup', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'running'. > 2019-12-03 11:15:46,893-08 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [] Finished polling Tasks, will poll again in 10 seconds. > 2019-12-03 11:15:50,059-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:15:50,064-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:15:58,074-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-76) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:15:58,080-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-76) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:16:08,138-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:16:08,144-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:16:18,219-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:16:18,227-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:16:28,297-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:16:28,304-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:16:38,371-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-42) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:16:38,377-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-42) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:16:48,389-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:16:48,394-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:16:58,406-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:16:58,412-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:17:08,423-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:17:08,429-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:17:18,442-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:17:18,449-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:17:28,461-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:17:28,466-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-99) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:17:38,476-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:17:38,481-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:17:48,492-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:17:48,499-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:17:58,509-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:17:58,515-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:18:08,577-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-68) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:18:08,582-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-68) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:18:18,651-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:18:18,657-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:18:28,723-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:18:28,728-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:18:38,739-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:18:38,744-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:18:48,754-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:18:48,759-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:18:58,821-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-51) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:18:58,826-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-51) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:19:08,889-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-79) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:19:08,894-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-79) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:19:18,906-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:19:18,911-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:19:28,925-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:19:28,932-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:19:38,945-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:19:38,952-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:19:48,962-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:19:48,967-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:19:58,978-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:19:58,983-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:20:09,063-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:20:09,068-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:20:19,154-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:20:19,159-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:20:29,169-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:20:29,174-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:20:39,184-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:20:39,189-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:20:49,199-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:20:49,205-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:20:59,217-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-30) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:20:59,222-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-30) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:21:09,233-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:21:09,239-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:21:19,254-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:21:19,260-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:21:29,324-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:21:29,332-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:21:39,341-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-6) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:21:39,347-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-6) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:21:49,413-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:21:49,418-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:21:59,429-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:21:59,434-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:22:09,497-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-17) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:22:09,502-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-17) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:22:19,568-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-82) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:22:19,573-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-82) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:22:29,583-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:22:29,589-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:22:39,600-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-11) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:22:39,606-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-11) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:22:49,616-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:22:49,621-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:22:59,632-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:22:59,638-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:23:09,648-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-71) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:23:09,653-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-71) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:23:19,663-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:23:19,669-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:23:29,679-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:23:29,684-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:23:39,694-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:23:39,699-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:23:49,712-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:23:49,719-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:23:59,781-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-4) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:23:59,786-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-4) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' (id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa') waiting on child command id: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' type:'CopyImageGroup' to complete > 2019-12-03 11:24:07,476-08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] SPMAsyncTask::PollTask: Polling task 'fe1a173f-869c-4522-9984-9c57bace7c87' (Parent Command 'CopyImageGroup', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. > 2019-12-03 11:24:07,479-08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] BaseAsyncTask::onTaskEndSuccess: Task 'fe1a173f-869c-4522-9984-9c57bace7c87' (Parent Command 'CopyImageGroup', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. > 2019-12-03 11:24:07,479-08 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] CommandAsyncTask::endActionIfNecessary: All tasks of command '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' has ended -> executing 'endAction' > 2019-12-03 11:24:07,479-08 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '0e3e3df9-73f5-4ca2-b592-0b9ce7371247'): calling endAction '. > 2019-12-03 11:24:07,480-08 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-3786044) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CopyImageGroup', > 2019-12-03 11:24:07,492-08 INFO [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command [id=0e3e3df9-73f5-4ca2-b592-0b9ce7371247]: Updating status to 'SUCCEEDED', The command end method logic will be executed by one of its parent commands. > 2019-12-03 11:24:07,492-08 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CopyImageGroup' completed, handling the result. > 2019-12-03 11:24:07,493-08 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CopyImageGroup' succeeded, clearing tasks. > 2019-12-03 11:24:07,493-08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fe1a173f-869c-4522-9984-9c57bace7c87' > 2019-12-03 11:24:07,496-08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='7bb7ac30-0406-4fbf-802f-57b1a575383f', ignoreFailoverLimit='false', taskId='fe1a173f-869c-4522-9984-9c57bace7c87'}), log id: 693daae4 > 2019-12-03 11:24:07,498-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, HSMClearTaskVDSCommand(HostName = KVMCLST1A, HSMTaskGuidBaseVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b', taskId='fe1a173f-869c-4522-9984-9c57bace7c87'}), log id: 3d8ab113 > 2019-12-03 11:24:07,514-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, HSMClearTaskVDSCommand, log id: 3d8ab113 > 2019-12-03 11:24:07,514-08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, SPMClearTaskVDSCommand, log id: 693daae4 > 2019-12-03 11:24:07,517-08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] BaseAsyncTask::removeTaskFromDB: Removed task 'fe1a173f-869c-4522-9984-9c57bace7c87' from DataBase > 2019-12-03 11:24:07,517-08 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-3786044) [d25a398d-f650-4911-ae2a-2ca2b54e8396] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '0e3e3df9-73f5-4ca2-b592-0b9ce7371247' > 2019-12-03 11:24:09,852-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'ExportOva' (id: 'd684210f-56b3-41ba-a699-e93341afdba2') waiting on child command id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' type:'CreateAllOvaDisks' to complete > 2019-12-03 11:24:09,858-08 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Command 'CreateAllOvaDisks' id: 'ef1b7dde-5908-4f1f-9d0a-228bf234d9fa' child commands '[0e3e3df9-73f5-4ca2-b592-0b9ce7371247]' executions were completed, status 'SUCCEEDED' > 2019-12-03 11:24:10,867-08 INFO [org.ovirt.engine.core.bll.storage.disk.CreateAllOvaDisksCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Ending command 'org.ovirt.engine.core.bll.storage.disk.CreateAllOvaDisksCommand' successfully. > 2019-12-03 11:24:10,877-08 INFO [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand' successfully. > 2019-12-03 11:24:10,887-08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, GetImageInfoVDSCommand( GetImageInfoVDSCommandParameters:{storagePoolId='7bb7ac30-0406-4fbf-802f-57b1a575383f', ignoreFailoverLimit='false', storageDomainId='c79208ed-8753-4ded-a137-8f08dfff5a40', imageGroupId='a1b75b90-04c8-4f8d-899e-7d0354e36b09', imageId='2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9'}), log id: 4ba602b2 > 2019-12-03 11:24:10,890-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, GetVolumeInfoVDSCommand(HostName = KVMCLST1A, GetVolumeInfoVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b', storagePoolId='7bb7ac30-0406-4fbf-802f-57b1a575383f', storageDomainId='c79208ed-8753-4ded-a137-8f08dfff5a40', imageGroupId='a1b75b90-04c8-4f8d-899e-7d0354e36b09', imageId='2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9'}), log id: 2524e570 > 2019-12-03 11:24:10,914-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, GetVolumeInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@4c8c88fe, log id: 2524e570 > 2019-12-03 11:24:10,914-08 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@4c8c88fe, log id: 4ba602b2 > 2019-12-03 11:24:10,988-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, PrepareImageVDSCommand(HostName = KVMCLST1A, PrepareImageVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b'}), log id: 11dbdafe > 2019-12-03 11:24:11,010-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 11dbdafe > 2019-12-03 11:24:11,014-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, GetQemuImageInfoVDSCommand(HostName = KVMCLST1A, GetVolumeInfoVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b', storagePoolId='7bb7ac30-0406-4fbf-802f-57b1a575383f', storageDomainId='c79208ed-8753-4ded-a137-8f08dfff5a40', imageGroupId='a1b75b90-04c8-4f8d-899e-7d0354e36b09', imageId='2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9'}), log id: 79697ffd > 2019-12-03 11:24:11,221-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, GetQemuImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.QemuImageInfo@5f543b3b, log id: 79697ffd > 2019-12-03 11:24:11,226-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.TeardownImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] START, TeardownImageVDSCommand(HostName = KVMCLST1A, ImageActionsVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b'}), log id: 1f4cb783 > 2019-12-03 11:24:11,230-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.TeardownImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [d25a398d-f650-4911-ae2a-2ca2b54e8396] FINISH, TeardownImageVDSCommand, return: StatusReturn:{status='Status [code=0, message=Done]'}, log id: 1f4cb783 > 2019-12-03 11:24:12,266-08 INFO [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Running command: CreateOvaCommand internal: true. > 2019-12-03 11:24:12,271-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] START, PrepareImageVDSCommand(HostName = KVMCLST1A, PrepareImageVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b'}), log id: 6e612ef8 > 2019-12-03 11:24:12,294-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 6e612ef8 > 2019-12-03 11:24:12,301-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] START, GetVolumeInfoVDSCommand(HostName = KVMCLST1A, GetVolumeInfoVDSCommandParameters:{hostId='960bcd30-c9b4-4aca-bc3e-4e710651d80b', storagePoolId='7bb7ac30-0406-4fbf-802f-57b1a575383f', storageDomainId='c79208ed-8753-4ded-a137-8f08dfff5a40', imageGroupId='a1b75b90-04c8-4f8d-899e-7d0354e36b09', imageId='2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9'}), log id: 648e4c38 > 2019-12-03 11:24:12,321-08 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] FINISH, GetVolumeInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@4c8c88fe, log id: 648e4c38 > 2019-12-03 11:24:12,332-08 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Executing Ansible command: [/usr/bin/ansible-playbook, --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa, --inventory=/tmp/ansible-inventory830646465207814253, --extra-vars=target_directory=/tmp, --extra-vars=ova_name=GRAYLOG.ova, --extra-vars=ovirt_ova_pack_ovf='<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><ovf:Envelope xmlns:ovf=\\\"http://schemas.dmtf.org/ovf/envelope/1\\\" xmlns:rasd=\\\"http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSet…" xmlns:vssd=\\\"http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingD…" xmlns:xsi=\\\"http://www.w3.org/2001/XMLSchema-instance\\\" xmlns=\\\"http://schemas.dmtf.org/ovf/envelope/1\\\" xmlns:ovirt=\\\"http://www.ovirt.org/ovf\\\"><References><File ovf:href=\\\"2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9\\\" ovf:id=\\\"2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9\\\" ovf:size=\\\"3448045568\\\"></File></References><NetworkSection><Info>List of networks</Info><Network ovf:name=\\\"net_HOME\\\"></Network></NetworkSection><DiskSection><Info>List of Virtual Disks</Info><Disk ovf:diskId=\\\"a1b75b90-04c8-4f8d-899e-7d0354e36b09\\\" ovf:capacity=\\\"100\\\" ovf:capacityAllocationUnits=\\\"byte * 2^30\\\" ovf:populatedSize=\\\"3448045568\\\" ovf:parentRef=\\\"\\\" ovf:fileRef=\\\"2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9\\\" ovf:format=\\\"http://www.gnome.org/~markmc/qcow-image-format.html\\\" ovf:volume-format=\\\"COW\\\" ovf:volume-type=\\\"Sparse\\\" ovf:disk-interface=\\\"VirtIO_SCSI\\\" ovf:boot=\\\"true\\\" ovf:pass-discard=\\\"false\\\" ovf:disk-alias=\\\"GRAYLOG_Disk1\\\" ovf:disk-description=\\\"GRAYLOG_Disk1\\\" ovf:wipe-after-delete=\\\"false\\\" ovf:description=\\\"Active VM\\\" ovf:disk_storage_type=\\\"IMAGE\\\" ovf:cinder_volume_type=\\\"\\\"></Disk></DiskSection><VirtualSystem ovf:id=\\\"aecb985a-450a-43f1-957a-09776aa48899\\\"><Name>GRAYLOG</Name><Description>GrayLog looging system</Description><CreationDate>2018/01/27 23:20:25</CreationDate><ExportDate>2019/12/03 19:24:12</ExportDate><DeleteProtected>false</DeleteProtected><SsoMethod>guest_agent</SsoMethod><IsSmartcardEnabled>false</IsSmartcardEnabled><TimeZone>Etc/GMT</TimeZone><default_boot_sequence>9</default_boot_sequence><Generation>66</Generation><ClusterCompatibilityVersion>4.2</ClusterCompatibilityVersion><VmType>1</VmType><ResumeBehavior>AUTO_RESUME</ResumeBehavior><MinAllocatedMem>4096</MinAllocatedMem><IsStateless>false</IsStateless><IsRunAndPause>false</IsRunAndPause><AutoStartup>true</AutoStartup><Priority>1</Priority><CreatedByUserId>5a655cbf-012a-0161-03dc-000000000307</CreatedByUserId><VmInit ovf:regenerateKeys=\\\"false\\\" ovf:networks=\\\"[ ]\\\"></VmInit><MigrationSupport>0</MigrationSupport><IsBootMenuEnabled>true</IsBootMenuEnabled><IsSpiceFileTransferEnabled>true</IsSpiceFileTransferEnabled><IsSpiceCopyPasteEnabled>true</IsSpiceCopyPasteEnabled><AllowConsoleReconnect>true</AllowConsoleReconnect><ConsoleDisconnectAction>LOCK_SCREEN</ConsoleDisconnectAction><CustomEmulatedMachine></CustomEmulatedMachine><CustomCpuName></CustomCpuName><PredefinedProperties></PredefinedProperties><UserDefinedProperties></UserDefinedProperties><MaxMemorySizeMb>16384</MaxMemorySizeMb><TemplateId>00000000-0000-0000-0000-000000000000</TemplateId><TemplateName>Blank</TemplateName><IsInitilized>true</IsInitilized><Origin>3</Origin><app_list>kernel-3.10.0-693.17.1.el7,ovirt-guest-agent-common-1.0.13-2.el7,kernel-3.10.0-693.el7</app_list><quota_id>5b0590e8-9611-4240-abdc-07cd4bbaed1f</quota_id><DefaultDisplayType>1</DefaultDisplayType><TrustedService>false</TrustedService><OriginalTemplateId>00000000-0000-0000-0000-000000000000</OriginalTemplateId><OriginalTemplateName>Blank</OriginalTemplateName><UseHostCpu>false</UseHostCpu><UseLatestVersion>false</UseLatestVersion><StopTime>2019/12/03 18:44:44</StopTime><OperatingSystemSection ovf:id=\\\"80\\\" ovirt:id=\\\"24\\\" ovf:required=\\\"false\\\"><Info>Guest Operating System</Info><Description>Red Hat Enterprise Linux 7.x x64</Description></OperatingSystemSection><VirtualHardwareSection><Info>2 CPU, 4096 Memory</Info><System><vssd:VirtualSystemType>ENGINE 4.1.0.0</vssd:VirtualSystemType></System><Item><rasd:Caption>2 virtual cpu</rasd:Caption><rasd:Description>Number of virtual CPU</rasd:Description><rasd:InstanceId>1</rasd:InstanceId><rasd:ResourceType>3</rasd:ResourceType><rasd:num_of_sockets>2</rasd:num_of_sockets><rasd:cpu_per_socket>1</rasd:cpu_per_socket><rasd:threads_per_cpu>1</rasd:threads_per_cpu><rasd:max_num_of_vcpus>16</rasd:max_num_of_vcpus><rasd:VirtualQuantity>2</rasd:VirtualQuantity></Item><Item><rasd:Caption>4096 MB of memory</rasd:Caption><rasd:Description>Memory Size</rasd:Description><rasd:InstanceId>2</rasd:InstanceId><rasd:ResourceType>4</rasd:ResourceType><rasd:AllocationUnits>MegaBytes</rasd:AllocationUnits><rasd:VirtualQuantity>4096</rasd:VirtualQuantity></Item><Item><rasd:Caption>GRAYLOG_Disk1</rasd:Caption><rasd:InstanceId>2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9</rasd:InstanceId><rasd:ResourceType>17</rasd:ResourceType><rasd:HostResource>ovf:disk/a1b75b90-04c8-4f8d-899e-7d0354e36b09</rasd:HostResource><rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent><rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template><rasd:ApplicationList></rasd:ApplicationList><rasd:StorageId>c79208ed-8753-4ded-a137-8f08dfff5a40</rasd:StorageId><rasd:StoragePoolId>7bb7ac30-0406-4fbf-802f-57b1a575383f</rasd:StoragePoolId><rasd:CreationDate>2019/12/03 19:15:38</rasd:CreationDate><rasd:LastModified>2019/12/03 19:15:38</rasd:LastModified><rasd:last_modified_date>2019/12/03 19:15:38</rasd:last_modified_date><Type>disk</Type><Device>disk</Device><rasd:Address>{type=drive, bus=0, controller=0, target=0, unit=0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>scsi0-0-0-0</Alias></Item><Item><rasd:Caption>Ethernet adapter on net_HOME</rasd:Caption><rasd:InstanceId>4adaa616-f98b-4e1f-b60f-14156665e344</rasd:InstanceId><rasd:ResourceType>10</rasd:ResourceType><rasd:OtherResourceType>net_HOME</rasd:OtherResourceType><rasd:ResourceSubType>3</rasd:ResourceSubType><rasd:Connection>net_HOME</rasd:Connection><rasd:Linked>true</rasd:Linked><rasd:Name>nic1</rasd:Name><rasd:ElementName>nic1</rasd:ElementName><rasd:MACAddress>00:1a:4a:16:01:04</rasd:MACAddress><rasd:speed>1000</rasd:speed><Type>interface</Type><Device>bridge</Device><rasd:Address>{type=pci, slot=0x03, bus=0x00, domain=0x0000, function=0x0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>net0</Alias></Item><Item><rasd:Caption>USB Controller</rasd:Caption><rasd:InstanceId>3</rasd:InstanceId><rasd:ResourceType>23</rasd:ResourceType><rasd:UsbPolicy>DISABLED</rasd:UsbPolicy></Item><Item><rasd:Caption>Graphical Controller</rasd:Caption><rasd:InstanceId>585fc912-16d6-421d-acda-eeaa17b955c1</rasd:InstanceId><rasd:ResourceType>20</rasd:ResourceType><rasd:VirtualQuantity>1</rasd:VirtualQuantity><rasd:SinglePciQxl>false</rasd:SinglePciQxl><Type>video</Type><Device>qxl</Device><rasd:Address>{type=pci, slot=0x02, bus=0x00, domain=0x0000, function=0x0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>video0</Alias><SpecParams><vgamem>16384</vgamem><heads>1</heads><vram>32768</vram><ram>65536</ram></SpecParams></Item><Item><rasd:Caption>Graphical Framebuffer</rasd:Caption><rasd:InstanceId>8c56ce64-dc7b-4a05-995a-eba2f31997fe</rasd:InstanceId><rasd:ResourceType>26</rasd:ResourceType><Type>graphics</Type><Device>spice</Device><rasd:Address></rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias></Alias></Item><Item><rasd:Caption>Graphical Framebuffer</rasd:Caption><rasd:InstanceId>ecfe54b9-26f1-4f44-9745-50cc45c9d50e</rasd:InstanceId><rasd:ResourceType>26</rasd:ResourceType><Type>graphics</Type><Device>vnc</Device><rasd:Address></rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias></Alias></Item><Item><rasd:Caption>CDROM</rasd:Caption><rasd:InstanceId>922be48e-a7ef-4490-8a46-df6d09d30aab</rasd:InstanceId><rasd:ResourceType>15</rasd:ResourceType><Type>disk</Type><Device>cdrom</Device><rasd:Address>{type=drive, bus=1, controller=0, target=0, unit=0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>true</IsReadOnly><Alias>ide0-1-0</Alias><SpecParams><path>CentOS-7-x86_64-NetInstall-1708.iso</path></SpecParams></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>e4f99c78-0a99-4b36-a3da-34e9267c9d59</rasd:InstanceId><Type>channel</Type><Device>spicevmc</Device><rasd:Address>{type=virtio-serial, bus=0, controller=0, port=3}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>channel2</Alias></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>0307995b-613a-4819-b667-4c077880eca9</rasd:InstanceId><Type>controller</Type><Device>ide</Device><rasd:Address>{type=pci, slot=0x01, bus=0x00, domain=0x0000, function=0x1}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>ide</Alias><SpecParams><index>0</index></SpecParams></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>0cc781c0-18c8-4096-8ab5-062eb51b1ef9</rasd:InstanceId><Type>channel</Type><Device>unix</Device><rasd:Address>{type=virtio-serial, bus=0, controller=0, port=1}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>channel0</Alias></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>55a35000-8756-45e5-89e6-32c914b1ef75</rasd:InstanceId><Type>channel</Type><Device>unix</Device><rasd:Address>{type=virtio-serial, bus=0, controller=0, port=2}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>channel1</Alias></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>c4cc5f36-26c3-4f1e-9a40-7cb1a4c28025</rasd:InstanceId><Type>controller</Type><Device>virtio-serial</Device><rasd:Address>{type=pci, slot=0x05, bus=0x00, domain=0x0000, function=0x0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>virtio-serial0</Alias></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>d64ea1bd-cf70-4786-9bfd-80121e30cbdf</rasd:InstanceId><Type>controller</Type><Device>usb</Device><rasd:Address>{type=pci, slot=0x01, bus=0x00, domain=0x0000, function=0x2}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>usb</Alias><SpecParams><index>0</index><model>piix3-uhci</model></SpecParams></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>2b5977d2-133d-4180-98b8-e96ab5401a0f</rasd:InstanceId><Type>rng</Type><Device>virtio</Device><rasd:Address>{type=pci, slot=0x06, bus=0x00, domain=0x0000, function=0x0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>rng0</Alias><SpecParams><source>urandom</source></SpecParams></Item><Item><rasd:ResourceType>0</rasd:ResourceType><rasd:InstanceId>f0c7a69b-6cd6-4428-84bf-5e862cb20202</rasd:InstanceId><Type>controller</Type><Device>virtio-scsi</Device><rasd:Address>{type=pci, slot=0x04, bus=0x00, domain=0x0000, function=0x0}</rasd:Address><BootOrder>0</BootOrder><IsPlugged>true</IsPlugged><IsReadOnly>false</IsReadOnly><Alias>scsi0</Alias></Item></VirtualHardwareSection></VirtualSystem></ovf:Envelope>', --extra-vars=ovirt_ova_pack_disks=/rhev/data-center/mnt/10.25.0.11:_mnt_vol01_kvmclst1_vmstore/c79208ed-8753-4ded-a137-8f08dfff5a40/images/a1b75b90-04c8-4f8d-899e-7d0354e36b09/2508c6d1-4ac2-4580-bb51-47fb3f4ca4f9::3448045568, /usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml] [Logfile: /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20191203112412-kvmclst1a.home.whootis.com-67d3d8f2.log] > 2019-12-03 11:24:22,637-08 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Ansible playbook command has exited with value: 2 > 2019-12-03 11:24:22,638-08 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Failed to create OVA. Please check logs for more details: /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20191203112412-kvmclst1a.home.whootis.com-67d3d8f2.log > 2019-12-03 11:24:22,640-08 ERROR [org.ovirt.engine.core.bll.exportimport.ExportOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Failed to create OVA file > 2019-12-03 11:24:22,640-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Command 'ExportOva' id: 'd684210f-56b3-41ba-a699-e93341afdba2' failed when attempting to perform the next operation, marking as FAILED '[ef1b7dde-5908-4f1f-9d0a-228bf234d9fa]' > 2019-12-03 11:24:22,640-08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [67d3d8f2] Command 'ExportOva' id: 'd684210f-56b3-41ba-a699-e93341afdba2' child commands '[ef1b7dde-5908-4f1f-9d0a-228bf234d9fa]' executions were completed, status 'FAILED' > 2019-12-03 11:24:23,655-08 ERROR [org.ovirt.engine.core.bll.exportimport.ExportOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Ending command 'org.ovirt.engine.core.bll.exportimport.ExportOvaCommand' with failure. > 2019-12-03 11:24:23,657-08 INFO [org.ovirt.engine.core.bll.exportimport.ExportOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [d25a398d-f650-4911-ae2a-2ca2b54e8396] Lock freed to object 'EngineLock:{exclusiveLocks='[aecb985a-450a-43f1-957a-09776aa48899=VM]', sharedLocks=''}' > 2019-12-03 11:24:23,667-08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [d25a398d-f650-4911-ae2a-2ca2b54e8396] EVENT_ID: IMPORTEXPORT_EXPORT_VM_TO_OVA_FAILED(1,225), Failed to export Vm GRAYLOG as a Virtual Appliance to path /tmp/GRAYLOG.ova on Host KVMCLST1A > > > Thanks for any help you can be! > -Geoff > >

1 0

Windows Sluggish Performance
by Vijay Sachdeva 03 Dec '19

03 Dec '19

Hello Everyone, I managed to install Windows Server 2016 server on Ovirt node using ovirt engine.All virtio-Win drivers also installed, but the performance of Windows using console is very sluggish and mouse pointer is not at all responsive. Can anyone let me know, what could be the reason. Not able to to even add IP to VM as mouse doesn’t work at all. Thanks

3 3

Re: Scenario: Ovirt-engine on hardware and recovery
by Jayme 03 Dec '19

03 Dec '19

Take a look at vProtect for backups. Works really well for our environment, and supports incremental backups as well which is very handy. It's free for up to 10 VMs to try. On Tue, Dec 3, 2019 at 4:20 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > Didi, > > Thanks for the reply. > > Yes, backups are done, I am just thinking in the case of Murphy's Law > where nothing works and the engine is gone. I am also new to oVirt and am > always loking for the "proper" backup method as you have laready mentioned > using engine-backup. > > Storage is NFS backed and backups are done in case of storage server > failure. > > What is the preferred VM backup method so both disk and VM config is > captured? Would rather not have to put agents on all the VM for OS level > backups and then have to manually recreate all of the machines. Do all the > VM configs get stored on the shared storage just like one of the other > virtualization vendors? > > Appreciate the time from folks on the list who respond. > ------------------------------ > *From:* Yedidyah Bar David <didi(a)redhat.com> > *Sent:* Tuesday, December 3, 2019 3:29 AM > *To:* Robert Webb <rwebb(a)ropeguru.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Scenario: Ovirt-engine on hardware and > recovery > > On Tue, Dec 3, 2019 at 4:38 AM <rwebb(a)ropeguru.com> wrote: > > > > So thinking about my setup, I am thinking through different failure > scenarios. > > > > So lets say I have a small physical server with 8 cores and 16GB RAM and > I install Centos 7 and ovirt-engine on bare metal. This would also be the > same scenario is the engine were on a VM. > > > > I run into a major hardware issue and completely lose the engine. > > You do take backups, right? :-) > > For the engine itself, you can use engine-backup. > > > > > How does one recover the cluster setup and not have to start from > scratch by having to rebuild all the nodes? Can the engine just be rebuilt > and the ovirt nodes be imported? > > You didn't mention your plans re storage. > > If you put your VMs on well-handled storage, and lose everything but > the storage, you can install a new engine, new nodes, and then import > the VMs from the existing storage. You will still miss some > configuration etc., but should mostly be ok. > > If you plan to use gluster storage using the same nodes, aka HCI, the > same applies, obviously, only that the nodes are your storage, and not > an external system. You still have to make sure you handle them > properly - backup, DR, etc., depending on your needs and budget. > > If you do have engine-backup backups, but lost all your storage, > engine-backup will not help you. > > > > > > This scenario is based on the nodes being built from the ovirt node iso. > > That's ok, but is not relevant. Main question is the storage. Can be > external or local/gluster with either ovirt-node or EL. > > Good luck, > > -- > Didi > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/M2URCFAZJQYCE… >

1 0

Re: Windows VirtIO drivers
by Vijay Sachdeva 03 Dec '19

03 Dec '19

Anyone, please let me know. Really stuck at this point. I am using Ovirt Manager 4.3 and while installing Windows VMs If I used disk type as IDE, I can see the disks to install the OS but if I use virtIO-scsi for disks the disk is not seen. So I attached floppy disk to VM, but it says no drivers detected. Can anyone let me know the best way to install Windows 2016? Thanks Vijay Sachdeva From: Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> Date: Tuesday, 3 December 2019 at 6:56 PM To: Gal Zaidman <gzaidman(a)redhat.com>, <users(a)ovirt.org> Subject: Re: [ovirt-users] Windows VirtIO drivers Hi Gal, I have downloaded the virtio-win_x86.vfd and uploaded to ISO domain. Used Run Once and attached the floppy drive, while looking for drivers in A drive it gives below error: I followed all the steps as per the guidelines shared in the document. Don’t know what I am missing. Thanks Vijay Sachdeva From: Gal Zaidman <gzaidman(a)redhat.com> Date: Tuesday, 3 December 2019 at 6:44 PM To: Vijay Sachdeva <vijay.sachdeva(a)indiqus.com>, <users(a)ovirt.org> Subject: Re: [ovirt-users] Windows VirtIO drivers Going back to the user list, just in case it will be relevant to others On Tue, Dec 3, 2019 at 3:02 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: Hi Gal, Thank you so much for replying, can you please a steps to do that. Right now what I am doing is: Installed Windows 2016 using IDE interface, but performance is very bad and it’ really hard to install any further .exe into it. I never used the IDE interface so I can't say from personal experience, but I don't think that should be the behavior, please open a bug on that so that we can examin. Other option I see attaching the floppy drive when we use “Run Once”, but problem is when pointing to floppy drive while loading drivers. It says no drivers detected. Using Run Once is the preferred option, see https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/ht… Make sure that you attach the latest virtio-win vdf (the link I sent earlier should be ok) and select the relevant drivers, if there is a problem then please open a bug on virtio-win project and send me the url so I will track it, thanks. Please share let me know the best way to perform this action as need to deliver the Ovirt infra to one of my customer and really stuck on this driver installation. Appreciate your quick response.!! Vijay Sachdeva From: Gal Zaidman <gzaidman(a)redhat.com> Date: Tuesday, 3 December 2019 at 6:22 PM To: Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> Subject: Re: [ovirt-users] Windows VirtIO drivers HI Vijay, There a package on fedora: virtio-win-0.1.173-2 which contains an MSI installer for the drivers and the ovirt agent (which is disabled on a normal installation, so please enable it when you install). This installer should replace the ovirt-wgt that we have currently on the next versions, and it is new, so please give us feedback once you used it. P.S the installer installs the drivers normally so if the regular drivers from virtio-win fail to install, then using the installer will not change much probably, and it means that you have a deeper problem. In case the installation fails then please generate logs but running the installer through cmd.exe "msiexec /i {installer} /l*v {path to log}" (https://www.advancedinstaller.com/user-guide/qa-log.html) and open a bug/reply here and I will try and help Thanks On Tue, Dec 3, 2019 at 2:11 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: Dear Team, How can we install virtIO windows drivers to Windows 2016. As of now I tried to attach floppy drive with virtio-win.vfd image but while installing drivers didn’t get loaded. Please guide or let me know the proper way to install drivers. Vijay Sachdeva _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XBYY54K62BYJ…

2 2

Zabbix monitoring within the node
by Vinícius Ferrão 03 Dec '19

03 Dec '19

Hello, There are some documentation on the web about using Zabbix with oVirt to monitor the hypervisor itself like here: https://github.com/hudecof/libzbxovirt and here: https://github.com/jensdepuydt/zabbix-ovirt But what are you ppl doing about this issue? Any recommendations on Zabbix monitoring with oVirt Node? Should I use the Zabbix Agent as a package or use SNMP instead? Thanks,

1 0

Re: Windows VirtIO drivers
by Vijay Sachdeva 03 Dec '19

03 Dec '19

Hi Gal, I have downloaded the virtio-win_x86.vfd and uploaded to ISO domain. Used Run Once and attached the floppy drive, while looking for drivers in A drive it gives below error: I followed all the steps as per the guidelines shared in the document. Don’t know what I am missing. Thanks Vijay Sachdeva From: Gal Zaidman <gzaidman(a)redhat.com> Date: Tuesday, 3 December 2019 at 6:44 PM To: Vijay Sachdeva <vijay.sachdeva(a)indiqus.com>, <users(a)ovirt.org> Subject: Re: [ovirt-users] Windows VirtIO drivers Going back to the user list, just in case it will be relevant to others On Tue, Dec 3, 2019 at 3:02 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: Hi Gal, Thank you so much for replying, can you please a steps to do that. Right now what I am doing is: Installed Windows 2016 using IDE interface, but performance is very bad and it’ really hard to install any further .exe into it. I never used the IDE interface so I can't say from personal experience, but I don't think that should be the behavior, please open a bug on that so that we can examin. Other option I see attaching the floppy drive when we use “Run Once”, but problem is when pointing to floppy drive while loading drivers. It says no drivers detected. Using Run Once is the preferred option, see https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/ht… Make sure that you attach the latest virtio-win vdf (the link I sent earlier should be ok) and select the relevant drivers, if there is a problem then please open a bug on virtio-win project and send me the url so I will track it, thanks. Please share let me know the best way to perform this action as need to deliver the Ovirt infra to one of my customer and really stuck on this driver installation. Appreciate your quick response.!! Vijay Sachdeva From: Gal Zaidman <gzaidman(a)redhat.com> Date: Tuesday, 3 December 2019 at 6:22 PM To: Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> Subject: Re: [ovirt-users] Windows VirtIO drivers HI Vijay, There a package on fedora: virtio-win-0.1.173-2 which contains an MSI installer for the drivers and the ovirt agent (which is disabled on a normal installation, so please enable it when you install). This installer should replace the ovirt-wgt that we have currently on the next versions, and it is new, so please give us feedback once you used it. P.S the installer installs the drivers normally so if the regular drivers from virtio-win fail to install, then using the installer will not change much probably, and it means that you have a deeper problem. In case the installation fails then please generate logs but running the installer through cmd.exe "msiexec /i {installer} /l*v {path to log}" (https://www.advancedinstaller.com/user-guide/qa-log.html) and open a bug/reply here and I will try and help Thanks On Tue, Dec 3, 2019 at 2:11 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: Dear Team, How can we install virtIO windows drivers to Windows 2016. As of now I tried to attach floppy drive with virtio-win.vfd image but while installing drivers didn’t get loaded. Please guide or let me know the proper way to install drivers. Vijay Sachdeva _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XBYY54K62BYJ…

2 1

Sigh... still vdsmd memory leak
by Chris Adams 03 Dec '19

03 Dec '19

I've seen vdsmd leak memory (RSS increasing) for a while, and never gotten anywhere with diagnosing it. I upgraded a small setup (2 hosts with hosted engine and iSCSI SAN) last week to 4.3.7 plus latest CentOS 7.7 updates, and I am still seeing this. Currently, there are only 3 active VMs, plus the engine, all running on one of the two node (which is also the SPM), and vdsmd on that node is up to 450M RSS (6 days after upgrading), which is up almost 100M in the last 24 hours. On the other node, vdsmd RSS has not changed in the last 24 hours (83M). I have a script watching the RSS on the growing node, and it's a pretty steady increase: 07:59:59.735 452080 +28 08:00:03.361 452104 +24 08:00:04.852 452112 +8 08:00:21.088 452116 +4 08:00:25.456 452120 +4 08:00:25.753 452124 +4 08:00:34.219 452132 +8 08:00:35.866 452136 +4 08:00:44.360 452140 +4 08:00:51.213 452144 +4 08:00:57.046 452152 +8 08:00:59.855 452160 +8 08:01:01.461 452164 +4 08:01:04.615 452168 +4 08:01:10.544 452172 +4 08:01:11.003 452180 +8 08:01:13.826 452184 +4 08:01:14.737 452188 +4 08:01:15.002 452192 +4 08:01:21.350 452196 +4 08:01:29.870 452200 +4 08:01:34.995 452208 +8 This is a problem I have in multiple oVirt setups - don't know if it is just something about how I installed or what, but I'd really like to work on getting this figured out and fixed. -- Chris Adams <cma(a)cmadams.net>

1 0

Scenario: Ovirt-engine on hardware and recovery
by rwebb＠ropeguru.com 03 Dec '19

03 Dec '19

So thinking about my setup, I am thinking through different failure scenarios. So lets say I have a small physical server with 8 cores and 16GB RAM and I install Centos 7 and ovirt-engine on bare metal. This would also be the same scenario is the engine were on a VM. I run into a major hardware issue and completely lose the engine. How does one recover the cluster setup and not have to start from scratch by having to rebuild all the nodes? Can the engine just be rebuilt and the ovirt nodes be imported? This scenario is based on the nodes being built from the ovirt node iso.

3 2

Re: Windows VirtIO drivers
by Gal Zaidman 03 Dec '19

03 Dec '19

Going back to the user list, just in case it will be relevant to others On Tue, Dec 3, 2019 at 3:02 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: > Hi Gal, > > > > Thank you so much for replying, can you please a steps to do that. Right > now what I am doing is: > > > > 1. Installed Windows 2016 using IDE interface, but performance is very > bad and it’ really hard to install any further .exe into it. > > I never used the IDE interface so I can't say from personal experience, but I don't think that should be the behavior, please open a bug on that so that we can examin. > > 1. > 2. Other option I see attaching the floppy drive when we use “Run > Once”, but problem is when pointing to floppy drive while loading drivers. > It says no drivers detected. > > Using Run Once is the preferred option, see https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/ht… Make sure that you attach the latest virtio-win vdf (the link I sent earlier should be ok) and select the relevant drivers, if there is a problem then please open a bug on virtio-win project and send me the url so I will track it, thanks. > > > Please share let me know the best way to perform this action as need to > deliver the Ovirt infra to one of my customer and really stuck on this > driver installation. > > > > Appreciate your quick response.!! > > > > Vijay Sachdeva > > > > > > *From: *Gal Zaidman <gzaidman(a)redhat.com> > *Date: *Tuesday, 3 December 2019 at 6:22 PM > *To: *Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> > *Subject: *Re: [ovirt-users] Windows VirtIO drivers > > > > HI Vijay, > > > > There a package on fedora: > > virtio-win-0.1.173-2 > > which contains an MSI installer for the drivers and the ovirt agent (which > is disabled on a normal installation, so please enable it when you install). > > > > This installer should replace the ovirt-wgt that we have currently on the > next versions, > > and it is new, so please give us feedback once you used it. > > > > P.S > > the installer installs the drivers normally so if the regular drivers from > virtio-win fail to install, then using the installer will not change much > probably, and it means that you have a deeper problem. > > In case the installation fails then please generate logs but running the > installer through cmd.exe "msiexec /i {installer} /l*v {path to log}" ( > https://www.advancedinstaller.com/user-guide/qa-log.html) > > and open a bug/reply here and I will try and help > > > > Thanks > > > > On Tue, Dec 3, 2019 at 2:11 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> > wrote: > > Dear Team, > > > > How can we install virtIO windows drivers to Windows 2016. As of now I > tried to attach floppy drive with virtio-win.vfd image but while installing > drivers didn’t get loaded. > > > > Please guide or let me know the proper way to install drivers. > > > > > > Vijay Sachdeva > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XBYY54K62BYJ… > >

1 0

Disk move succeed but didn't move content
by Juan Pablo Lorier 03 Dec '19

03 Dec '19

Hi, I've a fresh new install of ovirt 4.3 and tried to import an gluster vmstore. I managed to import via NFS the former data domain. The problem is that when I moved the disks of the vms to the new ISCSI data domain, I got a warning that sparse disk type will be converted to qcow2 disks, and after accepting, the disks were moved with no error. The problem is that the disks now figure as <1Gb size instead of the original size and thus, the vms fail to start. Is there any way to recover those disks? I have no backup of the vms :-( Regards

3 4

Current status of Ceph support in oVirt (2019)?
by victorhooi＠yahoo.com 03 Dec '19

03 Dec '19

Hi, I currently have a 3-node HA cluster running Proxmox (with integrated Ceph). oVirt looks pretty neat, however, and I'm excited to check it out. One of the things I love about Proxmox is the integrated Ceph support. I saw on the mailing lists that there is some talk of Ceph support earlier, but it was via OpenStack/Cinder. What exactly does this mean? 1. Does this require you to install OpenStack, or will a vanilla Ceph installation work? 2. Is it possible to deploy Ceph on the same nodes that run oVirt? (i.e. is a 3-node oVirt + Ceph cluster possible?) 3. Is there any monitoring/management of Ceph from within oVirt? (Guessing no?) 4. Are all the normal VM features working yet, or is this planned? 5. Is making Ceph a first-class citizen (like Gluster) on oVirt on the roadmap? Thanks, Victor https://www.reddit.com/r/ovirt/comments/ci38zp/ceph_rbd_support_in_ovirt_fo…

3 3

Windows VirtIO drivers
by Vijay Sachdeva 03 Dec '19

03 Dec '19

Dear Team, How can we install virtIO windows drivers to Windows 2016. As of now I tried to attach floppy drive with virtio-win.vfd image but while installing drivers didn’t get loaded. Please guide or let me know the proper way to install drivers. Vijay Sachdeva

1 0

oVirt nodes not responding
by Tim Herrmann 03 Dec '19

03 Dec '19

Hi everyone, we have an oVirt-Cluster with 5 nodes and 3 of them provide the storage with GlusterFS and replica 3. The cluster is running 87 VMs and has 9TB storage where 4TB is in use. The version of the oVirt-Engine is 4.1.8.2 and GlusterFS is 3.8.15. The servers are running in a HP bladecenter and are connected with 10GBit to each other. Actually we have some problems that all ovirt nodes periodically won't respond in the cluster with the following error messages in the oVirt webinterface: VDSM glustervirt05 command GetGlusterVolumeHealInfoVDS failed: Message timeout which can be caused by communication issues Host glustervirt05 is not responding. It will stay in Connecting state for a grace period of 68 seconds and after that an attempt to fence the host will be issued. Host glustervirt05 does not enforce SELinux. Current status: PERMISSIVE Executing power management status on Host glustervirt05 using Proxy Host glustervirt02 and Fence Agent ilo4:xxx.xxx.xxx.xxx. Manually synced the storage devices from host glustervirt05 Status of host glustervirt05 was set to Up. In the vdsm logfile I can find the following message: 2019-11-26 11:18:22,909+0100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {u'volumeName': u'data'}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.healInfo', 'id': u'2e86ed2c- 3e79-42c1-a7e4-c09bfbfc7794'} at 0x7fb938373190> timeout=60, duration=180 at 0x316a6d0> task#=2859802 at 0x1b70dd0> (executor:351) And I figured out, that the gluster heal info command takes very long: [root@glustervirt01 ~]# time gluster volume heal data info Brick glustervirt01:/gluster/data/brick1 Status: Connected Number of entries: 0 Brick glustervirt02:/gluster/data/brick1 Status: Connected Number of entries: 0 Brick glustervirt03:/gluster/data/brick2 Status: Connected Number of entries: 0 real 3m3.626s user 0m0.593s sys 0m0.559s A strange behavier is also that there is one virtual machine (a postgresql database) which stops running unexpectedly every one or two days ... The only thing that has been changed on the vm in the least past was a resize of the disk. VM replication-zabbix is down with error. Exit message: Lost connection with qemu process. And when we add or delete a larger disk with approximately 100GB in glusterfs, the glusterfs cluster freaks out won't respond anymore. This also results in paused VMs ... Has anyone an idea what could cause such problems?

2 1

Re: Scenario: Ovirt-engine on hardware and recovery
by Strahil 03 Dec '19

03 Dec '19

As this is not a VM, you won't have to worry about putting 1 host in maintenance. I make backups lile this: engine-backup --mode=backup --scope=all --file=/root/engine-bkp-empty-ovirt1-ver4.3.7-rc4--2019-11-30 --log=/var/log/engine-backup.log I'm specifying the host that will be used for restore, which is needed when the engine is VM. In your bare-metal case - you just need to recover the OS/harfware and restore from backup. Best Regards, Strahil NikolovOn Dec 3, 2019 04:36, rwebb(a)ropeguru.com wrote: > > So thinking about my setup, I am thinking through different failure scenarios. > > So lets say I have a small physical server with 8 cores and 16GB RAM and I install Centos 7 and ovirt-engine on bare metal. This would also be the same scenario is the engine were on a VM. > > I run into a major hardware issue and completely lose the engine. How does one recover the cluster setup and not have to start from scratch by having to rebuild all the nodes? Can the engine just be rebuilt and the ovirt nodes be imported? > > This scenario is based on the nodes being built from the ovirt node iso. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XBFYDGZNCT4VL…

1 0

Re: host failed to attach one of the storage domains attached to it
by Strahil 03 Dec '19

03 Dec '19

Hi Daniel, The hosts go to unresponsive mode as one of the gluster volumes cannot be mounted. Verify that all host can mount (over /mnt) all gluster volumes. If they do, just set the hosts into maintenance and then activate them. Best Regards, Strahil NikolovOn Dec 2, 2019 12:03, Daniel Menzel <daniel.menzel(a)menzel-it.net> wrote: > > Dear all, > > after a GlusterFS reboot of two servers out of three (and thus a loss of quorum) by an admin we've got strange problems: > > The hosted engine and one export work perfectly fine. (Same three servers!) > Another export (main one) seems to be fine within GlusterFS itself. But: When we activate this domain most of the hosts go into "non operational" with the subject's error message. As soon as we deactivate this domain all those hosts come back online. Strange thing: The GlusterFS export seems to be mounted on the SPM. > > Does anyone know what could have happened and how to fix that? > > Kind regards > Daniel

1 0

Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
by Strahil 03 Dec '19

03 Dec '19

Hi Jiffin, Thanks for the info. As I'm now on Gluster v7 , I hope it won't happen again. It's nice to know it got fixed. Best Regards, Strahil NikolovOn Dec 2, 2019 11:32, Jiffin Thottan <jthottan(a)redhat.com> wrote: > > Hi Krutika, > > Apparently, in context acl info got corrupted see brick logs > > [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: > client: > CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > gfid: be318638-e8a0-4c6d-977d-7a937aa84806, > req(uid:107,gid:107,perm:1,ngrps:4), > ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) > [Permission denied] > > which resulted in the situation. There was one bug similar was reported https://bugzilla.redhat.com/show_bug.cgi?id=1668286 and > > it got fixed in 6.6 release IMO https://review.gluster.org/#/c/glusterfs/+/23233/. But here he mentioned he saw the issue when > > he upgraded from 6.5 to 6.6 > > One way to workaround is to perform a dummy setfacl(preferably using root) on the corrupted files which will forcefully fetch the acl > > info again from backend and update the context. Another approach to restart brick process(kill and vol start force) > > Regards, > Jiffin > > ----- Original Message ----- > From: "Krutika Dhananjay" <kdhananj(a)redhat.com> > To: "Strahil Nikolov" <hunter86_bg(a)yahoo.com>, "Jiffin Thottan" <jthottan(a)redhat.com>, "raghavendra talur" <rtalur(a)redhat.com> > Cc: "Nir Soffer" <nsoffer(a)redhat.com>, "Rafi Kavungal Chundattu Parambil" <rkavunga(a)redhat.com>, "users" <users(a)ovirt.org>, "gluster-user" <gluster-users(a)gluster.org> > Sent: Monday, December 2, 2019 11:48:22 AM > Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing > > Sorry about the late response. > > I looked at the logs. These errors are originating from posix-acl > translator - > > > > *[2019-11-17 07:55:47.090065] E [MSGID: 115050] > [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496: > LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6 > (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6), > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > error-xlator: data_fast-access-control [Permission denied][2019-11-17 > 07:55:47.090174] I [MSGID: 139001] > [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > gfid: be318638-e8a0-4c6d-977d-7a937aa84806, > req(uid:36,gid:36,perm:1,ngrps:3), > ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) > [Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050] > [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497: > LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7 > (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7), > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > error-xlator: data_fast-access-control [Permission denied][2019-11-17 > 07:55:47.090299] I [MSGID: 139001] > [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > gfid: be318638-e8a0-4c6d-977d-7a937aa84806, > req(uid:36,gid:36,perm:1,ngrps:3), > ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) > [Permission denied]* > > Jiffin/Raghavendra Talur, > Can you help? > > -Krutika > > On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> > wrote: > > > Hi Nir,All, > > > > it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached > > screenshot of oVirt running on v7 gluster). > > It seems strange that both my serious issues with oVirt are related to > > gluster issue (1st gluster v3 to v5 migration and now this one). > > > > I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all > > nodes. > > Now both Engine and all my VMs are back online - so if you hit issues with > > 6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before > > deciding to wipe everything. > > > > @Krutika, > > > > I guess you will ask for the logs, so let's switch to gluster-users about > > this one ? > > > > Best Regards, > > Strahil Nikolov > > > > В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov < > > hunter86_bg(a)yahoo.com> написа: > > > > > > Hi Krutika, > > > > I have enabled TRACE log level for the volume data_fast, > > > > but the issue is not much clear: > > FUSE reports: > > > > [2019-11-25 21:31:53.478130] I [MSGID: 133022] > > [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of > > gfid=6d9ed2e5-d4f2-4749-839b-2f1 > > 3a68ed472 from backend > > [2019-11-25 21:32:43.564694] W [MSGID: 114031] > > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: > > remote operation failed. Path:

1 0

Re: NFS Storage Domain on OpenMediaVault
by Amit Bawer 03 Dec '19

03 Dec '19

On Mon, Dec 2, 2019 at 7:21 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > Thanks for the response. > > As it turned out, the issue was on the export where I had to remove > subtree_check and added no_root_squash and it started working. > > Being new to the setup, I am still trying to work through some config > issues and deciding if I want to continue. The main thing I am looking for > is good HA and failover capability. Been using Proxmox VE and I like the > admin of it, but failover still needs work. Simple things like auto > migration when rebooting a host does not exist and is something I need. > for RHEV, you can check the Cluster scheduling policies https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/ht… > Robert > ------------------------------ > *From:* Strahil Nikolov <hunter86_bg(a)yahoo.com> > *Sent:* Sunday, December 1, 2019 2:54 PM > *To:* users(a)ovirt.org <users(a)ovirt.org>; Robert Webb <rwebb(a)ropeguru.com> > *Subject:* Re: [ovirt-users] NFS Storage Domain on OpenMediaVault > > Does sanlock user has rights on the ...../dom_md/ids ? > > Check the sanlock.service for issues. > journalctl -u sanlock.service > > Best Regards, > Strahil Nikolov > > В неделя, 1 декември 2019 г., 17:22:21 ч. Гринуич+2, rwebb(a)ropeguru.com < > rwebb(a)ropeguru.com> написа: > > > I have a clean install with openmediavault as backend NFS and cannot get > it to work. Keep getting permission errors even though I created a vdsm > user and kvm group; and they are the owners of the directory on OMV with > full permissions. > > The directory gets created on the NFS side for the host, but then get the > permission error and is removed form the host but the directory structure > is left on the NFS server. > > Logs: > > From the engine: > > Error while executing action New NFS Storage Domain: Unexpected exception > > From the oVirt node log: > > 2019-11-29 10:03:02 136998 [30025]: open error -13 EACCES: no permission > to open /rhev/data-center/mnt/192.168.1.56: > _export_Datastore-oVirt/f38b19e4-8060-4467-860b-09cf606ccc15/dom_md/ids > 2019-11-29 10:03:02 136998 [30025]: check that daemon user sanlock 179 > group sanlock 179 has access to disk or file. > > File system on Openmediavault: > > drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 . > drwxr-xr-x 9 root root 4096 Nov 27 20:56 .. > drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 > f38b19e4-8060-4467-860b-09cf606ccc15 > > drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 . > drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 .. > drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 dom_md > drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 images > > drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 . > drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 .. > -rw-rw----+ 1 vdsm kvm 0 Nov 29 10:03 ids > -rw-rw----+ 1 vdsm kvm 16777216 Nov 29 10:03 inbox > -rw-rw----+ 1 vdsm kvm 0 Nov 29 10:03 leases > -rw-rw-r--+ 1 vdsm kvm 343 Nov 29 10:03 metadata > -rw-rw----+ 1 vdsm kvm 16777216 Nov 29 10:03 outbox > -rw-rw----+ 1 vdsm kvm 1302528 Nov 29 10:03 xleases > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/ILKNT57F6VUHE… > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/KH2H2V254Z7T7… >

2 1

NFS Storage Domain on OpenMediaVault
by rwebb＠ropeguru.com 02 Dec '19

02 Dec '19

I have a clean install with openmediavault as backend NFS and cannot get it to work. Keep getting permission errors even though I created a vdsm user and kvm group; and they are the owners of the directory on OMV with full permissions. The directory gets created on the NFS side for the host, but then get the permission error and is removed form the host but the directory structure is left on the NFS server. Logs: From the engine: Error while executing action New NFS Storage Domain: Unexpected exception From the oVirt node log: 2019-11-29 10:03:02 136998 [30025]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/192.168.1.56:_export_Datastore-oVirt/f38b19e4-8060-4467-860b-09cf606ccc15/dom_md/ids 2019-11-29 10:03:02 136998 [30025]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file. File system on Openmediavault: drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 . drwxr-xr-x 9 root root 4096 Nov 27 20:56 .. drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 f38b19e4-8060-4467-860b-09cf606ccc15 drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 . drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 .. drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 dom_md drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 images drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 . drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 .. -rw-rw----+ 1 vdsm kvm 0 Nov 29 10:03 ids -rw-rw----+ 1 vdsm kvm 16777216 Nov 29 10:03 inbox -rw-rw----+ 1 vdsm kvm 0 Nov 29 10:03 leases -rw-rw-r--+ 1 vdsm kvm 343 Nov 29 10:03 metadata -rw-rw----+ 1 vdsm kvm 16777216 Nov 29 10:03 outbox -rw-rw----+ 1 vdsm kvm 1302528 Nov 29 10:03 xleases

4 3

host failed to attach one of the storage domains attached to it
by Daniel Menzel 02 Dec '19

02 Dec '19

Dear all, after a GlusterFS reboot of two servers out of three (and thus a loss of quorum) by an admin we've got strange problems: 1. The hosted engine and one export work perfectly fine. (Same three servers!) 2. Another export (main one) seems to be fine within GlusterFS itself. But: When we activate this domain most of the hosts go into "non operational" with the subject's error message. As soon as we deactivate this domain all those hosts come back online. Strange thing: The GlusterFS export seems to be mounted on the SPM. Does anyone know what could have happened and how to fix that? Kind regards Daniel

1 0

Overt Networking VM not pinging
by Vijay Sachdeva 02 Dec '19

02 Dec '19

Dear Community, I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working. Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem? Any help would be highly appreciated. Thanks Vijay Sachdeva Senior Manager – Service Delivery IndiQus Technologies O +91 11 4055 1411 | M +91 8826699409 www.indiqus.com

3 7

hyperconverged single node with SSD cache fails gluster creation
by thomas＠hoberg.net 01 Dec '19

01 Dec '19

I am seeing more success than failures at creating single and triple node hyperconverged setups after some weeks of experimentation so I am branching out to additional features: In this case the ability to use SSDs as cache media for hard disks. I tried first with a single node that combined caching and compression and that fails during the creation of LVMs. I tried again without the VDO compression, but actually the results where identical whilst VDO compression but without the LV cache worked ok. I tried various combinations, using less space etc., but the results are always the same and unfortunately rather cryptic (substituted the physical disk label with {disklabel}): TASK [gluster.infra/roles/backend_setup : Extend volume group] ***************** failed: [{hostname}] (item={u'vgname': u'gluster_vg_{disklabel}p1', u'cachethinpoolname': u'gluster_thinpool_gluster_vg_{disklabel}p1', u'cachelvname': u'cachelv_gluster_thinpool_gluster_vg_{disklabel}p1', u'cachedisk': u'/dev/sda4', u'cachemetalvname': u'cache_gluster_thinpool_gluster_vg_{disklabel}p1', u'cachemode': u'writeback', u'cachemetalvsize': u'70G', u'cachelvsize': u'630G'}) => {"ansible_loop_var": "item", "changed": false, "err": " Physical volume \"/dev/mapper/vdo_{disklabel}p1\" still in use\n", "item": {"cachedisk": "/dev/sda4", "cachelvname": "cachelv_gluster_thinpool_gluster_vg_{disklabel}p1", "cachelvsize": "630G", "cachemetalvname": "cache_gluster_thinpool_gluster_vg_{disklabel}p1", "cachemetalvsize": "70G", "cachemode": "writeback", "cachethinpoolname": "gluster_thinpool_gluster_vg_{disklabel}p1", "vgname": "gluster_vg_{disklabel}p1"}, "msg": "Unable to reduce gluster_vg_{disklabel}p1 by /dev/dm-15.", "rc": 5} somewhere within that I see something that points to a race condition ("still in use"). Unfortunately I have not been able to pinpoint the raw logs which are used at that stage and I wasn't able to obtain more info. At this point quite a bit of storage setup is already done, so rolling back for a clean new attempt, can be a bit complicated, with reboots to reconcile the kernel with data on disk. I don't actually believe it's related to single node and I'd be quite happy to move the creation of the SSD cache to a later stage, but in a VDO setup, this looks slightly complex to someone without intimate knowledge of LVS-with-cache-and-perhaps-thin/VDO/Gluster all thrown into one. Needless the feature set (SSD caching & compressed-dedup) sounds terribly attractive but when things don't just work, it's more terrifying.

5 7

Moving HostedEngine
by Joseph Goldman 28 Nov '19

28 Nov '19

Hi List, In one of my installs, I set up the first storage domain (and where the HostedEngine is) on a bigger NFS NAS - since then I have created a Gluster volume that spans the 3 hosts and I'm putting a few VM's in there for higher reliability (as SAN is single point of failure) namely I'd like to put HostedEngine in there so it stays up no matter what and can help report if issues occur (network issue to NAS, NAS dies etc etc) Looking through other posts and documentation, there's no real way to move the HostedEngine storage, is this correct? The solution I've seen is to backup the hosted engine DB, blow it away, and re-deploy it from the .backup file configuring it to the new storage domain in the deploy script - is this the only process? How likely is this to fail? Is it likely that all VM's and settings will be picked straight back up and continue to operate like normal? I dont have a test setup to play around with atm so just trying to gauge confidence in such a solution. Thanks, Joe

3 3

Migrate VM from oVirt to oVirt
by adrianquintero＠gmail.com 28 Nov '19

28 Nov '19

Hello, What would be the procedure to migrate a VM from oVirt to oVirt? Migrate from oVirt 4.2 running on Site A to oVirt 4.3 Site B. thanks!

5 7

Re: HostedEngine Deployment fails on AMD EPYC 7402P 4.3.7
by Strahil 28 Nov '19

28 Nov '19

The HostedEngine does it automatically, but the option that defines the OVF refresh interval is not accurate. Just power it up without that option (cluster in maintenance) and keep it running for a day. On the next day power it off and try to power it up via hosted-engine --vm-start. Best Regards, Strahil NikolovOn Nov 28, 2019 11:54, Ralf Schenk <rs(a)databay.de> wrote: > > Hello, > > I did something like that via "virsh edit HostedEngine". > > But how is the change written back to the shared storage ("hosted_storage" so it stays permanent for HA Engine ? > > I was able to boot up HostedEngine manually via virsh start after removing the required flag from XML (I first added a user to sasldb in /etc/libvirt/passwd.db to be able to log into libvirt). > > Bye > > > Am 28.11.2019 um 05:51 schrieb Strahil: > > Hi Ralf, > When the deployment fail - you can dump the xml from virsh , edit it, undefine the current HostedEngine and define your modified HostedEngine 's xml. > > Once you do that, you can try to start the > VM. > > Good luck. > > Best Regards, > Strahil Nikolov > > On Nov 27, 2019 18:28, Ralf Schenk <rs(a)databay.de> wrote: >> >> Hello, >> >> This week I tried to deploy Hosted Engine on Ovirt-Node-NG 4.3.7 based Host. >> >> At the time the locally deployed Engine ist copied to hosted-storage (in my case NFS) and deployment tries to start the Engine (via ovirt-ha-agent) this fails. >> >> QUEMU Log (/var/log/libvirt/qemu/HostedEngine.log) only shows "2019-11-27 16:17:16.833+0000: shutting down, reason=failed". >> >> Researching the cause is: The built Libvirt VM XML includes the feature "virt-ssbd" as requirement, which is simly not there. >> >> From VM XML: >> >> <cpu mode='custom' match='exact' check='partial'> >> <model fallback='allow'>EPYC</model> >> <topology sockets='16' cores='4' threads='1'/> >> <feature policy='require' name='ibpb'/> >> <feature policy='require' name='virt-ssbd'/> >> >> from cat /proc/cpuinfo: >> >> processor : 47 >> vendor_id : AuthenticAMD >> cpu family : 23 >> model : 49 >> model name : AMD EPYC 7402P 24-Core Processor >> stepping : 0 >> microcode : 0x830101c >> cpu MHz : 2800.000 >> cache size : 512 KB >> physical id : 0 >> siblings : 48 >> core id : 30 >> cpu cores : 24 >> api

1 0

Re: spice connection error
by Strahil 28 Nov '19

28 Nov '19

Check on host 'node3' in the Network configuration if you have a network out of sync or any clues about the error . Best Regards, Strahil NikolovOn Nov 28, 2019 11:44, Kim Kargaard <Kim.Kargaard(a)noroff.no> wrote: > > Hi, > > I am getting the attached error when trying to move the Display from the one network to the other within the cluster. I can't see any place to set an IP for the vnic that I want to set as the display network. > > Any thoughts? > > Kim > > On 28/11/2019, 06:41, "Strahil" <hunter86_bg(a)yahoo.com> wrote: > > As far as I know , the engine plays a role as a proxy during the establishment of the connection. > Check that you can reach both engine and the host from your system. > > For the same reason, I use noVNC - as you just need a single port to the engine in addition to the rest of the settings. > > Best Regards, > Strahil NikolovOn Nov 27, 2019 11:27, kim.kargaard(a)noroff.no wrote: > > > > Hi, > > > > When trying to connect from a remote network on the spice console to a VM, I get the following error: > > > > (remote-viewer:80195): virt-viewer-WARNING **: 11:05:22.322: Channel error: Could not connect to proxy server xx.xx.xx.xx: Socket I/O timed out > > > > I found that the display is set to the management network and not the VM networkn in the cluster logical network. However, when I try to set the other vlan to be the display network, I get the following error: > > > > Error while executing action: Cannot edit Network. IP address has to be set for the NIC that bears a role network. Network: student-vlan100, Nic: p2p1.100 on host node3 violates that rule. > > > > I am not sure what this means. Any ideas? > > > > Kind regards > > > > Kim > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6YEI3I4NCAHQO… > >

1 0

VDSM Errors see below
by rob.downer＠orbitalsystems.co.uk 28 Nov '19

28 Nov '19

I have removed all deployment of the hosted engine by running the following commands. ovirt-hosted-engine-cleanup vdsm-tool configure --force systemctl restart libvirtd systemctl restart vdsm on my hosts I have the following ovirt 1 is the host I ran the hosted engine setup. I have set the Gluster network to use the same subnet and set up forward and reverse DNS for the Gluster port network NIC's I had this working using a separate subnet but thought to try it on the same subnet to avoid any issues that may have occurred while using a separate network subnet. the main Host IP address is still showing in Unamanaged Connections on Ovirt 1 .. is this anything to be concerned about after running the commands above... I have restarted all machines. All come back with these VDSM errors... Node 1 [root@ovirt1 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: inactive (dead) since Thu 2019-11-28 13:29:40 UTC; 37min ago Process: 31178 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 30721 ExecStart=/usr/share/vdsm/daemonAdapter -0 /dev/null -1 /dev/null -2 /dev/null /usr/share/vdsm/vdsmd (code=exited, status=0/SUCCESS) Main PID: 30721 (code=exited, status=0/SUCCESS) Nov 26 22:42:49 ovirt1.kvm.private vdsm[30721]: WARN MOM not available, KSM stats will be missing. Nov 26 22:42:49 ovirt1.kvm.private vdsm[30721]: WARN Not ready yet, ignoring event '|virt|VM_status|871ce9d5-417a-4278-8446-28b681760c1b' args={'871ce9d5-417a-4278-8446-28b681760c1b': {'status': 'Poweri... Nov 28 13:28:43 ovirt1.kvm.private vdsm[30721]: WARN File: /var/run/vdsm/trackedInterfaces/eno2 already removed Nov 28 13:29:26 ovirt1.kvm.private vdsm[30721]: WARN File: /var/lib/libvirt/qemu/channels/871ce9d5-417a-4278-8446-28b681760c1b.com.redhat.rhevm.vdsm already removed Nov 28 13:29:26 ovirt1.kvm.private vdsm[30721]: WARN File: /var/lib/libvirt/qemu/channel/target/domain-1-HostedEngineLocal/org.qemu.guest_agent.0 already removed Nov 28 13:29:39 ovirt1.kvm.private vdsm[30721]: WARN MOM not available. Nov 28 13:29:39 ovirt1.kvm.private vdsm[30721]: WARN MOM not available, KSM stats will be missing. Nov 28 13:29:39 ovirt1.kvm.private systemd[1]: Stopping Virtual Desktop Server Manager... Nov 28 13:29:39 ovirt1.kvm.private vdsmd_init_common.sh[31178]: vdsm: Running run_final_hooks Nov 28 13:29:40 ovirt1.kvm.private systemd[1]: Stopped Virtual Desktop Server Manager. Hint: Some lines were ellipsized, use -l to show in full. [root@ovirt1 ~]# nodectl check Status: WARN Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... BAD NODE 2 [root@ovirt2 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-28 13:57:30 UTC; 1min 13s ago Process: 3626 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 5418 (vdsmd) Tasks: 38 CGroup: /system.slice/vdsmd.service └─5418 /usr/bin/python2 /usr/share/vdsm/vdsmd Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: libvirt: Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing' Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running dummybr Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running tune_system Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running test_space Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running test_lo Nov 28 13:57:30 ovirt2.kvm.private systemd[1]: Started Virtual Desktop Server Manager. Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN File: /var/run/vdsm/trackedInterfaces/eno1 already removed Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN File: /var/run/vdsm/trackedInterfaces/eno2 already removed Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN MOM not available. Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN MOM not available, KSM stats will be missing. [root@ovirt2 ~]# nodectl check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... OK [root@ovirt2 ~]# Node 3 [root@ovirt3 ~]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-28 13:57:08 UTC; 12min ago Process: 2849 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 3220 (vdsmd) Tasks: 38 CGroup: /system.slice/vdsmd.service └─3220 /usr/bin/python2 /usr/share/vdsm/vdsmd Nov 28 13:57:07 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running nwfilter Nov 28 13:57:07 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running dummybr Nov 28 13:57:08 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running tune_system Nov 28 13:57:08 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running test_space Nov 28 13:57:08 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running test_lo Nov 28 13:57:08 ovirt3.kvm.private systemd[1]: Started Virtual Desktop Server Manager. Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN File: /var/run/vdsm/trackedInterfaces/eno1 already removed Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN File: /var/run/vdsm/trackedInterfaces/eno2 already removed Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN MOM not available. Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN MOM not available, KSM stats will be missing. [root@ovirt3 ~]# nodectl check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... OK [root@ovirt3 ~]#

1 0

Re: HostedEngine Deployment fails on AMD EPYC 7402P 4.3.7
by Strahil 28 Nov '19

28 Nov '19

Hi Ralf, When the deployment fail - you can dump the xml from virsh , edit it, undefine the current HostedEngine and define your modified HostedEngine 's xml. Once you do that, you can try to start the VM. Good luck. Best Regards, Strahil NikolovOn Nov 27, 2019 18:28, Ralf Schenk <rs(a)databay.de> wrote: > > Hello, > > This week I tried to deploy Hosted Engine on Ovirt-Node-NG 4.3.7 based Host. > > At the time the locally deployed Engine ist copied to hosted-storage (in my case NFS) and deployment tries to start the Engine (via ovirt-ha-agent) this fails. > > QUEMU Log (/var/log/libvirt/qemu/HostedEngine.log) only shows "2019-11-27 16:17:16.833+0000: shutting down, reason=failed". > > Researching the cause is: The built Libvirt VM XML includes the feature "virt-ssbd" as requirement, which is simly not there. > > From VM XML: > > <cpu mode='custom' match='exact' check='partial'> > <model fallback='allow'>EPYC</model> > <topology sockets='16' cores='4' threads='1'/> > <feature policy='require' name='ibpb'/> > <feature policy='require' name='virt-ssbd'/> > > from cat /proc/cpuinfo: > > processor : 47 > vendor_id : AuthenticAMD > cpu family : 23 > model : 49 > model name : AMD EPYC 7402P 24-Core Processor > stepping : 0 > microcode : 0x830101c > cpu MHz : 2800.000 > cache size : 512 KB > physical id : 0 > siblings : 48 > core id : 30 > cpu cores : 24 > apicid : 61 > initial apicid : 61 > fpu : yes > fpu_exception : yes > cpuid level : 16 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl xtopology nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb cat_l3 cdp_l3 hw_pstate sme retpoline_amd ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip overflow_recov succor smca > bogomips : 5600.12 > TLB size : 3072 4K pages > clflush size : 64 > cache_alignment : 64 > address sizes : 43 bits physical, 48 bits virtual > power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14] > > Any solution/workaround available ?

2 1

Re: Moving HostedEngine
by Strahil 28 Nov '19

28 Nov '19

It is highly 'super duper' not recommended to do that. Also, consider using thinLVM (requirement for gluster snapshots) for a brick of the separate gluster volume. I'm using the 3rd partirion of my OS SSD to host my gluster's volume and I have no issues so far. Best Regards, Strahil NikolovOn Nov 28, 2019 01:05, Joseph Goldman <joseph(a)goldman.id.au> wrote: > > So I can't host OTHER VM's on this gluster volume? If its already a running GLuser for other VMs i can't now re-reploy HE in that gluster volume? > > On 2019-11-28 3:27 AM, Alan G wrote: > > I've had to do this a couple of times and always ended up with a working system in the end. > > As a fall back option (although I've never had to use it) I have a backup engine VM running completely outside of oVIrt (ESXi host in my case). Then if the hosted_engine deploy fails for any reason you can restore onto the backup vm as a temp solution while you work through the hosted engine deploy issues. > > A few things that come to mind: - > > * You will need a dedicated gluster volume for hosted_storage and it needs to be replica+arbiter. >> >> * Make sure you put the cluster in global maint mode before performing the engine backup, I recall having issues with the restore when I didn't do that. >> * Migrate all other VMs off the host running Engine before doing the backup. This will be the host you will restore onto. >> >> >> ---- On Wed, 27 Nov 2019 09:46:23 +0000 Joseph Goldman <joseph(a)goldman.id.au> wrote ---- >> >>> Hi List, >>> >>> In one of my installs, I set up the first storage domain (and where >>> the HostedEngine is) on a bigger NFS NAS - since then I have created a >>> Gluster volume that spans the 3 hosts and I'm putting a few VM's in >>> there for higher reliability (as SAN is single point of failure) namely >>> I'd like to put HostedEngine in there so it stays up no matter what and >>> can help report if issues occur (network issue to NAS, NAS dies etc etc) >>> >>> Looking through other posts and documentation, there's no real way to >>> move the HostedEngine storage, is this correct? The solution I've seen >>> is to backup the hosted engine DB, blow it away, and re-deploy it from >>> the .backup file configuring it to the new storage domain in the deploy >>> script - is this the only process? How likely is this to fail? Is it >>> likely that all VM's and settings will be picked straight back up and >>> continue to operate like normal? I dont have a test setup to play around >>> with atm so just trying to gauge confidence in such a solution. >>> >>> Thanks, >>> Joe >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct:

1 0

Re: Moving HostedEngine
by Strahil 28 Nov '19

28 Nov '19

Hi Joseph, It is mandatory to create a dedicated gluster volume for the HostedEngine. There is a thread which explains why and the risks of not following these guidlines. You can set the volume on thinLVM, so you can snapshot your gluster's volume before patching of the engine (of course you have to shutdown the engine in advance). About the transfer, the safest way is to backup & restore as you are not migrating from 1 type of storage domain to another of same type. Yet, I can't help you with that procedure - as I have never done it. Best Regards, Strahil NikolovOn Nov 27, 2019 11:46, Joseph Goldman <joseph(a)goldman.id.au> wrote: > > Hi List, > > In one of my installs, I set up the first storage domain (and where > the HostedEngine is) on a bigger NFS NAS - since then I have created a > Gluster volume that spans the 3 hosts and I'm putting a few VM's in > there for higher reliability (as SAN is single point of failure) namely > I'd like to put HostedEngine in there so it stays up no matter what and > can help report if issues occur (network issue to NAS, NAS dies etc etc) > > Looking through other posts and documentation, there's no real way to > move the HostedEngine storage, is this correct? The solution I've seen > is to backup the hosted engine DB, blow it away, and re-deploy it from > the .backup file configuring it to the new storage domain in the deploy > script - is this the only process? How likely is this to fail? Is it > likely that all VM's and settings will be picked straight back up and > continue to operate like normal? I dont have a test setup to play around > with atm so just trying to gauge confidence in such a solution. > > Thanks, > Joe > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZTO6Q3UTEDJCD…

1 0

Re: spice connection error
by Strahil 28 Nov '19

28 Nov '19

As far as I know , the engine plays a role as a proxy during the establishment of the connection. Check that you can reach both engine and the host from your system. For the same reason, I use noVNC - as you just need a single port to the engine in addition to the rest of the settings. Best Regards, Strahil NikolovOn Nov 27, 2019 11:27, kim.kargaard(a)noroff.no wrote: > > Hi, > > When trying to connect from a remote network on the spice console to a VM, I get the following error: > > (remote-viewer:80195): virt-viewer-WARNING **: 11:05:22.322: Channel error: Could not connect to proxy server xx.xx.xx.xx: Socket I/O timed out > > I found that the display is set to the management network and not the VM networkn in the cluster logical network. However, when I try to set the other vlan to be the display network, I get the following error: > > Error while executing action: Cannot edit Network. IP address has to be set for the NIC that bears a role network. Network: student-vlan100, Nic: p2p1.100 on host node3 violates that rule. > > I am not sure what this means. Any ideas? > > Kind regards > > Kim > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6YEI3I4NCAHQO…

1 0

Re: Engine deployment last step.... Can anyone help ?
by Strahil 28 Nov '19

28 Nov '19

Hi Rob, Can you check the gluster's volume ? gluster volume status engine gluster volume heal engine info summary Still, this is strange that the firewall is having such differences. Fastest way to sync that node to the settings from the other is to: 1. Backup /etc/firewalld 2. Remove /etc/firewalld 3. Copy /etc/firewalld from another node (either scp or rsync wil do) 4. Restart the firewalld daemon 5. Make a test firewalld reload Best Regards, Strahil NikolovOn Nov 27, 2019 00:09, Rob <rob.downer(a)orbitalsystems.co.uk> wrote: > > Is this the issue ? > > on the deployment host the firewall is active but only on eno2, eno1 appears to have decided to be unmanaged … > > also the deployment host has 17 rules active > > the other two have 7 each… > > > Zone > Interfaces > IP Range > Public > default > eno2 > * > > Unmanaged Interfaces > > > Name > > IP Address > > Sending > > Receiving > > ;vdsmdummy; > > > Inactive > > eno1 > > > > > ovirtmgmt > > 192.168.100.38/24 > > > > virbr0-nic > > >> >> On 25 Nov 2019, at 17:16, Parth Dhanjal <dparth(a)redhat.com> wrote: >> >> Rather than disabling firewalld, >> You can add the ports and restart the firewalld service >> >> # firewall-cmd --add-service=cockpit >> >> # firewall-cmd --add-service=cockpit --permanent >> >> >> >> On Mon, Nov 25, 2019 at 10:43 PM Amit Bawer <abawer(a)redhat.com> wrote: >>> >>> firewalld is running? >>> >>> systemctl disable --now firewalld >>> >>> On Monday, November 25, 2019, Rob <rob.downer(a)orbitalsystems.co.uk> wrote: >>>> >>>> It can’t be DNS, since the engine runs on a separate network anyway ie front end, so why can’t it reach the Volume I wonder >>>> >>>> >>>>> On 25 Nov 2019, at 12:55, Gobinda Das <godas(a)redhat.com> wrote: >>>>> >>>>> There could be two reason >>>>> 1- Your gluster service may not be running. >>>>> 2- In Storage Connection there mentioned <volumename> may not exists >>>>> >>>>> can you please paste the output of "gluster volume status" ? >>>>> >>>>> On Mon, Nov 25, 2019 at 5:03 PM Rob <rob.downer(a)orbitalsystems.co.uk> wrote: >>>>>> >>>>>> [ INFO ] skipping: [localhost] >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Add glusterfs storage domain] >>>>>> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400. >>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Failed to fetch Gluster Volume List]\". HTTP response code is 400.”} >>>>>> >>>>>> >>>>>>> On 25 Nov 2019, at 09:16, Rob <rob.downer(a)orbitalsystems.co.uk> wrote: >>>>>>> >>>>>>> Yes, >>>>>>> >>>>>>> I’ll restart all Nodes after wiping the failed setup of Hosted engine using. >>>>>>> >>>>>>> ovirt-hosted-engine-cleanup >>>>>>> vdsm-tool configure --force >>>>>>> systemctl restart libvirtd >>>>>>> systemctl restart vdsm >>>>>>> >>>>>>> although last time I did >>>>>>> >>>>>>> systemctl restart vdsm >>>>>>> >>>>>>> VDSM did not restart maybe that is OK as Hosted Engine was then de deployed or is that the issue ? >>>>>>> >>>>>>> >>>>>>>> On 25 Nov 2019, at 09:13, Parth Dhanjal <dparth(a)redhat.com> wrote: >>>>>>>> >>>>>>>> Can you please share the error in case it fails again? >>>>>>>> >>>>>>>> On Mon, Nov 25, 2019 at 2:42 PM Rob <rob.downer(a)orbitalsystems.co.uk> wrote: >>>>>>>>> >>>>>>>>> hmm, I’’l try again, that failed last time. >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 25 Nov 2019, at 09:08, Parth Dhanjal <dparth(a)redhat.com> wrote: >>>>>>>>>> >>>>>>>>>> Hey! >>>>>>>>>> >>>>>>>>>> For >>>>>>>>>> Storage Connection you can add - <hostname1>:/engine >>>>>>>>>> And for >>>>>>>>>> Mount Options - backup-volfile-servers=<hostname2>:<hostname3> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Nov 25, 2019 at 2:31 PM <rob.downer(a)orbitalsystems.co.uk> wrote: >>>>>>>>>>> >>>>>>>>>>> So... >>>>>>>>>>> >>>>>>>>>>> I have got to the last step >>>>>>>>>>> >>>>>>>>>>> 3 Machines with Gluster Storage configured however at the last screen >>>>>>>>>>> >>>>>>>>>>> ....Deploying the Engine to Gluster and the wizard does not auto fill the two fields >>>>>>>>>>> >>>>>>>>>>> Hosted Engine Deployment >>>>>>>>>>> >>>>>>>>>>> Storage Connection >>>>>>>>>>> and >>>>>>>>>>> Mount Options >>>>>>>>>>> >>>>>>>>>>> I also had to expand /tmp as it was not big enough to fit the engine before moving... >>>>>>>>>>> >>>>>>>>>>> What can I do to get the auto complete sorted out ? >>>>>>>>>>> >>>>>>>>>>> I have tried entering ovirt1.kvm.private:/gluster_lv_engine - The Volume name >>>>>>>>>>> and >>>>>>>>>>> ovirt1.kvm.private:/gluster_bricks/engine >>>>>>>>>>> >>>>>>>>>>> Ovirt1 being the actual machine I'm running this on. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NL4HS6MIKWQAG… >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LQ5YXX7IVV6ZP… >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VHQ3LTES6ABPL… >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> Thanks, >>>>> Gobinda >>>> >>>> >

1 0

HostedEngine Deployment fails on AMD EPYC 7402P 4.3.7
by Ralf Schenk 27 Nov '19

27 Nov '19

Hello, This week I tried to deploy Hosted Engine on Ovirt-Node-NG 4.3.7 based Host. At the time the locally deployed Engine ist copied to hosted-storage (in my case NFS) and deployment tries to start the Engine (via ovirt-ha-agent) this fails. QUEMU Log (/var/log/libvirt/qemu/HostedEngine.log) only shows "2019-11-27 16:17:16.833+0000: shutting down, reason=failed". Researching the cause is: The built Libvirt VM XML includes the feature "virt-ssbd" as requirement, which is simly not there. From VM XML: <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>EPYC</model> <topology sockets='16' cores='4' threads='1'/> <feature policy='require' name='ibpb'/> <feature policy='require' name='virt-ssbd'/> from cat /proc/cpuinfo: processor : 47 vendor_id : AuthenticAMD cpu family : 23 model : 49 model name : AMD EPYC 7402P 24-Core Processor stepping : 0 microcode : 0x830101c cpu MHz : 2800.000 cache size : 512 KB physical id : 0 siblings : 48 core id : 30 cpu cores : 24 apicid : 61 initial apicid : 61 fpu : yes fpu_exception : yes cpuid level : 16 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl xtopology nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb cat_l3 cdp_l3 hw_pstate sme retpoline_amd ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip overflow_recov succor smca bogomips : 5600.12 TLB size : 3072 4K pages clflush size : 64 cache_alignment : 64 address sizes : 43 bits physical, 48 bits virtual power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14] Any solution/workaround available ? -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------

1 0

Certificate of host is invalid
by Jon bae 27 Nov '19

27 Nov '19

Hello everybody, since last update to 4.3.7 I get this error message: Certificate of host *host.name <http://host.name>* is invalid. The certificate doesn't contain valid subject alternative name, please enroll new certificate for the host. Have you an idea of how I can fix that? Regards Jonathan

1 0

spice connection error
by kim.kargaard＠noroff.no 27 Nov '19

27 Nov '19

Hi, When trying to connect from a remote network on the spice console to a VM, I get the following error: (remote-viewer:80195): virt-viewer-WARNING **: 11:05:22.322: Channel error: Could not connect to proxy server xx.xx.xx.xx: Socket I/O timed out I found that the display is set to the management network and not the VM networkn in the cluster logical network. However, when I try to set the other vlan to be the display network, I get the following error: Error while executing action: Cannot edit Network. IP address has to be set for the NIC that bears a role network. Network: student-vlan100, Nic: p2p1.100 on host node3 violates that rule. I am not sure what this means. Any ideas? Kind regards Kim

1 0

Engine deployment last step.... Can anyone help ?
by rob.downer＠orbitalsystems.co.uk 26 Nov '19

26 Nov '19

So... I have got to the last step 3 Machines with Gluster Storage configured however at the last screen ....Deploying the Engine to Gluster and the wizard does not auto fill the two fields Hosted Engine Deployment Storage Connection and Mount Options I also had to expand /tmp as it was not big enough to fit the engine before moving... What can I do to get the auto complete sorted out ? I have tried entering ovirt1.kvm.private:/gluster_lv_engine - The Volume name and ovirt1.kvm.private:/gluster_bricks/engine Ovirt1 being the actual machine I'm running this on. Thanks

5 16

Re: Engine deployment last step.... Can anyone help ?
by Strahil 26 Nov '19

26 Nov '19

You need to set it also permanent via '--permanent', otherwise after a reload - it's gone. Best Regards, Strahil NikolovOn Nov 25, 2019 19:51, Rob <rob.downer(a)orbitalsystems.co.uk> wrote: > > [root@ovirt1 ~]# firewall-cmd --add-service=cockpit > Warning: ALREADY_ENABLED: 'cockpit' already in 'public' > success > [root@ovirt1 ~]# > > > >> On 25 Nov 2019, at 17:16, Parth Dhanjal <dparth(a)redhat.com> wrote: >> >> Rather than disabling firewalld, >> You can add the ports and restart the firewalld service >> >> # firewall-cmd --add-service=cockpit >> >> # firewall-cmd --add-service=cockpit --permanent >> >> >> >> On Mon, Nov 25, 2019 at 10:43 PM Amit Bawer <abawer(a)redhat.com> wrote: >>> >>> firewalld is running? >>> >>> systemctl disable --now firewalld >>> >>> On Monday, November 25, 2019, Rob <rob.downer(a)orbitalsystems.co.uk> wrote: >>>> >>>> It can’t be DNS, since the engine runs on a separate network anyway ie front end, so why can’t it reach the Volume I wonder >>>> >>>> >>>>> On 25 Nov 2019, at 12:55, Gobinda Das <godas(a)redhat.com> wrote: >>>>> >>>>> There could be two reason >>>>> 1- Your gluster service may not be running. >>>>> 2- In Storage Connection there mentioned <volumename> may not exists >>>>> >>>>> can you please paste the output of "gluster volume status" ? >>>>> >>>>> On Mon, Nov 25, 2019 at 5:03 PM Rob <rob.downer(a)orbitalsystems.co.uk> wrote: >>>>>> >>>>>> [ INFO ] skipping: [localhost] >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Add glusterfs storage domain] >>>>>> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400. >>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Failed to fetch Gluster Volume List]\". HTTP response code is 400.”} >>>>>> >>>>>> <

2 1

Cannot obtain information from export domain
by Arthur Rodrigues Stilben 26 Nov '19

26 Nov '19

Hello everyone, I'm trying to export a virtual machine, but I'm getting the following error: 2019-11-26 16:30:06,250-02 ERROR [org.ovirt.engine.core.bll.exportimport.GetVmsFromExportDomainQuery] (default task-22) [b9a0b9d5-2127-4002-9cee-2e3525bccc89] Exception: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to GetVmsInfoVDS, error = Storage domain does not exist: (u'5ac6c35d-0406-4a06-a682-ed8fb2d1933f',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2019-11-26 16:30:06,249-02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-22) [b9a0b9d5-2127-4002-9cee-2e3525bccc89] EVENT_ID: IMPORTEXPORT_GET_VMS_INFO_FAILED(200), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to retrieve VM/Templates information from export domain BackupMV The version of the oVirt that I am using is 4.1. Att, -- Arthur Rodrigues Stilben

1 0

Re: Fail to attach/create disk from template when running via Ansible
by Vrgotic, Marko 26 Nov '19

26 Nov '19

Hi Martin, Resending as delivery of my last email failed. From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> Date: Monday, 25 November 2019 at 09:53 To: Martin Perina <mperina(a)redhat.com>, Eyal Shenitzky <eshenitz(a)redhat.com> Cc: Ryan Barry <rbarry(a)redhat.com>, Ondra Machacek <omachace(a)redhat.com>, "users(a)ovirt.org" <users(a)ovirt.org>, Darko Stojchev <D.Stojchev(a)activevideo.com>, Tal Nisan <tnisan(a)redhat.com> Subject: Re: [ovirt-users] Re: Fail to attach/create disk from template when running via Ansible Hi Martin, Thank you for driving this issue. I have updated the bug ticket on bugzilla, as I was able to reproduce it via WebUI as well. Regarding my other question, How to clean up Locked Instances”, not to lose sight from it: I have attached the engine and server log as requested (below), but I did not get any answer yet. If more information is needed to proceed, please let me know. Kindly awaiting your reply. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo From: Martin Perina <mperina(a)redhat.com> Date: Thursday, 21 November 2019 at 10:00 To: Eyal Shenitzky <eshenitz(a)redhat.com> Cc: Ryan Barry <rbarry(a)redhat.com>, Ondra Machacek <omachace(a)redhat.com>, "users(a)ovirt.org" <users(a)ovirt.org>, Darko Stojchev <D.Stojchev(a)activevideo.com>, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com>, Tal Nisan <tnisan(a)redhat.com> Subject: Re: [ovirt-users] Re: Fail to attach/create disk from template when running via Ansible On Thu, Nov 21, 2019 at 9:17 AM Martin Perina <mperina(a)redhat.com<mailto:mperina@redhat.com>> wrote: Hi Eyal, could you please take a look? There is NPE cause most probably by passing invalid vdsId from CreateSnapshotCommand ... Just noticed that Marko created https://bugzilla.redhat.com/show_bug.cgi?id=1772853 to track the issue so let's continue the investigation there ... Thanks, Martin On Thu, Nov 21, 2019 at 8:51 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi Martin, The engine.log and server.log files attached. Here is the list of servers from this morning, in last hour, if it helps with going through the logs. [cid:16e8d08b4114cff311] The ones from previous screenshot are there since few days ago. If any additional info or tests are needed, let me know. Kindly awaiting your reply. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo From: Martin Perina <mperina(a)redhat.com<mailto:mperina@redhat.com>> Date: Thursday, 21 November 2019 at 08:34 To: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>>, Ryan Barry <rbarry(a)redhat.com<mailto:rbarry@redhat.com>> Cc: Ondra Machacek <omachace(a)redhat.com<mailto:omachace@redhat.com>>, "users(a)ovirt.org<mailto:users@ovirt.org>" <users(a)ovirt.org<mailto:users@ovirt.org>>, Darko Stojchev <D.Stojchev(a)activevideo.com<mailto:D.Stojchev@activevideo.com>> Subject: Re: [ovirt-users] Re: Fail to attach/create disk from template when running via Ansible Hi Marko, could you please share your engine.log and server.log? Thanks, Martin On Thu, Nov 21, 2019 at 8:31 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi Martin and Ondra, I cannot find clean way to remove the entities which are in unknown (probably locked) state. Due to the issue I reported below, the entities I try to create end up in, I assume, locked state. [A screenshot of a cell phone Description automatically generated] Due to this state, I cannot remove them from UI, and if I try to remove them using Ansible, get following error: FAILED! => {"changed": false, "msg": "Timeout exceed while waiting on result state of the entity." Would you be so kind to provide me with steps to clean up these locked instances? Kindly awaiting your reply. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> Date: Friday, 15 November 2019 at 11:34 To: Martin Perina <mperina(a)redhat.com<mailto:mperina@redhat.com>> Cc: Ondra Machacek <omachace(a)redhat.com<mailto:omachace@redhat.com>>, "users(a)ovirt.org<mailto:users@ovirt.org>" <users(a)ovirt.org<mailto:users@ovirt.org>>, Darko Stojchev <D.Stojchev(a)activevideo.com<mailto:D.Stojchev@activevideo.com>> Subject: Re: [ovirt-users] Re: Fail to attach/create disk from template when running via Ansible Hi Martin, Here is the BUG URI: https://bugzilla.redhat.com/show_bug.cgi?id=1772853 I am not sure if I selected correct component and team, but its created. If any additional information is required, do not hesitate to let me know. Kindly awaiting your reply. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo From: Martin Perina <mperina(a)redhat.com<mailto:mperina@redhat.com>> Date: Friday, 15 November 2019 at 11:07 To: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> Cc: Ondra Machacek <omachace(a)redhat.com<mailto:omachace@redhat.com>>, "users(a)ovirt.org<mailto:users@ovirt.org>" <users(a)ovirt.org<mailto:users@ovirt.org>>, Darko Stojchev <D.Stojchev(a)activevideo.com<mailto:D.Stojchev@activevideo.com>> Subject: Re: [ovirt-users] Re: Fail to attach/create disk from template when running via Ansible On Fri, Nov 15, 2019 at 11:05 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi Martin, You mean to create a bug under Ansible? Sure. No, in bugzilla for oVirt, it doesn't seem to me as ansible modules bug, but rather some issue in engine. Thanks, Martin ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo From: Martin Perina <mperina(a)redhat.com<mailto:mperina@redhat.com>> Date: Friday, 15 November 2019 at 10:30 To: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> Cc: Ondra Machacek <omachace(a)redhat.com<mailto:omachace@redhat.com>>, "users(a)ovirt.org<mailto:users@ovirt.org>" <users(a)ovirt.org<mailto:users@ovirt.org>>, Darko Stojchev <D.Stojchev(a)activevideo.com<mailto:D.Stojchev@activevideo.com>> Subject: Re: [ovirt-users] Re: Fail to attach/create disk from template when running via Ansible Hi Marko, could you please create a bug for that and attach full SOS collection logs from engine and relevant host? Thanks, Martin On Wed, Nov 13, 2019 at 2:54 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi Ondra, I have rebuilt the staging with 4.3.6 and still same issue. Images are there, I can again create VM via UI, but not via Ansible. Here is the output of the complete run form engine.log: 2019-11-12 12:09:00,457Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-49) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2019-11-12 12:09:00,978Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-49) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2019-11-12 12:09:01,538Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-49) [233de27] Running command: CreateUserSessionCommand internal: false. 2019-11-12 12:09:01,549Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-49) [233de27] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '172.16.105.206' using session 'bsgNpfGJTNtzuuQWSyXVJqHhpLK4TY9TQRERWxxzGtpHonoqdk7WIHGZ7mID2+QkBKH9bPp8HVhyvDu66tzPnQ==' logged in. 2019-11-12 12:09:01,691Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:01,773Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Running command: AddVmCommand internal: false. Entities affected : ID: d3c90169-ecc8-46a0-a4ec-7c3d99b81b61 Type: ClusterAction group CREATE_VM with role type USER, ID: 819aecf6-c83d-4470-9915-e309390d8098 Type: VmTemplateAction group CREATE_VM with role type USER, ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: StorageAction group CREATE_DISK with role type USER 2019-11-12 12:09:01,866Z INFO [org.ovirt.engine.core.bll.AddRngDeviceCommand] (default task-49) [4d4703a1] Running command: AddRngDeviceCommand internal: true. Entities affected : ID: 6343c55a-c67e-4d75-a767-dfc9412006e6 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2019-11-12 12:09:01,876Z INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-49) [4d4703a1] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='6343c55a-c67e-4d75-a767-dfc9412006e6', status='ImageLocked', exitStatus='Normal'}), log id: 4eedcefb 2019-11-12 12:09:01,887Z INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-49) [4d4703a1] FINISH, SetVmStatusVDSCommand, return: , log id: 4eedcefb 2019-11-12 12:09:01,906Z INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Running command: CreateSnapshotFromTemplateCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:01,928Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] START, CreateVolumeVDSCommand( CreateVolumeVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', imageSizeInBytes='8589934592', volumeFormat='COW', newImageId='7270dd54-a075-4b14-aca3-7c2a118057f0', imageType='Sparse', newImageDescription='', imageInitialSizeInBytes='0', imageId='767c6837-5191-4650-a49d-abca7d69a708', sourceImageGroupId='400688e8-3c5b-4cca-9ef8-bbb6c79d54ed'}), log id: 2dfac68e 2019-11-12 12:09:02,009Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] FINISH, CreateVolumeVDSCommand, return: 7270dd54-a075-4b14-aca3-7c2a118057f0, log id: 2dfac68e 2019-11-12 12:09:02,014Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' 2019-11-12 12:09:02,014Z INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandMultiAsyncTasks::attachTask: Attaching task '0eb09758-113f-43b9-aa67-97d2e5a57e05' to command 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b'. 2019-11-12 12:09:02,027Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Adding task '0eb09758-113f-43b9-aa67-97d2e5a57e05' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2019-11-12 12:09:02,098Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] EVENT_ID: USER_ADD_VM_STARTED(37), VM centos-testvm-a910 creation was initiated by admin@internal-authz. 2019-11-12 12:09:02,098Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] BaseAsyncTask::startPollingTask: Starting to poll task '0eb09758-113f-43b9-aa67-97d2e5a57e05'. 2019-11-12 12:09:02,758Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command 'AddVm' (id: '97d4103b-acef-4b55-a379-71a1baa04a4f') waiting on child command id: 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' type:'CreateSnapshotFromTemplate' to complete 2019-11-12 12:09:04,773Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command 'AddVm' (id: '97d4103b-acef-4b55-a379-71a1baa04a4f') waiting on child command id: 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' type:'CreateSnapshotFromTemplate' to complete 2019-11-12 12:09:06,117Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now 2019-11-12 12:09:06,121Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Failed in 'HSMGetAllTasksStatusesVDS' method 2019-11-12 12:09:06,128Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt-staging-hv-02.avinity.tv<http://ovirt-staging-hv-02.avinity.tv> command HSMGetAllTasksStatusesVDS failed: Invalid parameter: 'capacity=8589934592' 2019-11-12 12:09:06,128Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] SPMAsyncTask::PollTask: Polling task '0eb09758-113f-43b9-aa67-97d2e5a57e05' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2019-11-12 12:09:06,131Z ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] BaseAsyncTask::logEndTaskFailure: Task '0eb09758-113f-43b9-aa67-97d2e5a57e05' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'capacity=8589934592', code = 100', -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'capacity=8589934592', code = 100' 2019-11-12 12:09:06,133Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' has ended -> executing 'endAction' 2019-11-12 12:09:06,133Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b'): calling endAction '. 2019-11-12 12:09:06,133Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CreateSnapshotFromTemplate', 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command [id=fb915b7e-4814-4e93-a3ef-4f40c4e3c41b]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands. 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' completed, handling the result. 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' succeeded, clearing tasks. 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '0eb09758-113f-43b9-aa67-97d2e5a57e05' 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', taskId='0eb09758-113f-43b9-aa67-97d2e5a57e05'}), log id: baaf58a 2019-11-12 12:09:06,138Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] START, HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-02.avinity.tv<http://ovirt-staging-hv-02.avinity.tv>, HSMTaskGuidBaseVDSCommandParameters:{hostId='3c70c563-e32e-4352-b1e1-f6bcc41f77e2', taskId='0eb09758-113f-43b9-aa67-97d2e5a57e05'}), log id: 7321c57d 2019-11-12 12:09:06,147Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] FINISH, HSMClearTaskVDSCommand, return: , log id: 7321c57d 2019-11-12 12:09:06,147Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] FINISH, SPMClearTaskVDSCommand, return: , log id: baaf58a 2019-11-12 12:09:06,152Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] BaseAsyncTask::removeTaskFromDB: Removed task '0eb09758-113f-43b9-aa67-97d2e5a57e05' from DataBase 2019-11-12 12:09:06,152Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' 2019-11-12 12:09:08,790Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command 'AddVm' id: '97d4103b-acef-4b55-a379-71a1baa04a4f' child commands '[fb915b7e-4814-4e93-a3ef-4f40c4e3c41b]' executions were completed, status 'FAILED' 2019-11-12 12:09:09,821Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:09,825Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:09,835Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:09,873Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 28915db3 2019-11-12 12:09:09,907Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:09,907Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:09,907Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] FINISH, DeleteImageGroupVDSCommand, return: , log id: 28915db3 2019-11-12 12:09:09,907Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:09,918Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Removed task '25bf448d-9271-4a85-ad9b-9c63012753ac' from DataBase 2019-11-12 12:09:09,983Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] transaction rolled back 2019-11-12 12:09:09,983Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:10,001Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:10,002Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:12,019Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:12,023Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:12,041Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:12,063Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 376f65dc 2019-11-12 12:09:12,090Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:12,090Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:12,090Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] FINISH, DeleteImageGroupVDSCommand, return: , log id: 376f65dc 2019-11-12 12:09:12,090Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:12,101Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Removed task 'ebe0e879-8595-44db-9d64-e237afd6c8e8' from DataBase 2019-11-12 12:09:12,157Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] transaction rolled back 2019-11-12 12:09:12,157Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:12,157Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:12,157Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:12,157Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:12,172Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:12,172Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:16,192Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:16,195Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:16,215Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:16,240Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 75027448 2019-11-12 12:09:16,263Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:16,263Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:16,263Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] FINISH, DeleteImageGroupVDSCommand, return: , log id: 75027448 2019-11-12 12:09:16,263Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:16,274Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Removed task '688274fc-0596-44b5-b167-36c86b8b8e5d' from DataBase 2019-11-12 12:09:16,341Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] transaction rolled back 2019-11-12 12:09:16,341Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:16,341Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:16,342Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:16,342Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:16,357Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:16,357Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:24,376Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:24,380Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:24,387Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:24,419Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 29a560b1 2019-11-12 12:09:24,444Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:24,444Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:24,444Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] FINISH, DeleteImageGroupVDSCommand, return: , log id: 29a560b1 2019-11-12 12:09:24,445Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:24,455Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Removed task '13b2490d-09a6-4230-a5d3-7b86b536e8c3' from DataBase 2019-11-12 12:09:24,509Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] transaction rolled back 2019-11-12 12:09:24,509Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:24,509Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:24,509Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:24,509Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:24,527Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:24,527Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:34,547Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:34,551Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:34,568Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:34,591Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 207e5265 2019-11-12 12:09:34,617Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:34,617Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:34,618Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] FINISH, DeleteImageGroupVDSCommand, return: , log id: 207e5265 2019-11-12 12:09:34,618Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:34,629Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Removed task 'ad58e8d8-13cc-4ae4-8785-9df252b38c1f' from DataBase 2019-11-12 12:09:34,688Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] transaction rolled back 2019-11-12 12:09:34,688Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:34,688Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:34,688Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:34,688Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:34,707Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:34,707Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:42,871Z INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-75) [58fc7cd2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[02a18cd9-ead2-44c7-9923-8fe84d1865ee=PROVIDER]', sharedLocks=''}' 2019-11-12 12:09:42,880Z INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-75) [58fc7cd2] Running command: SyncNetworkProviderCommand internal: true. 2019-11-12 12:09:42,975Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-49) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2019-11-12 12:09:43,096Z INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-75) [58fc7cd2] Lock freed to object 'EngineLock:{exclusiveLocks='[02a18cd9-ead2-44c7-9923-8fe84d1865ee=PROVIDER]', sharedLocks=''}' 2019-11-12 12:09:44,729Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:44,732Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:44,757Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:44,780Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: aaea0bc 2019-11-12 12:09:44,804Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:44,804Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:44,804Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] FINISH, DeleteImageGroupVDSCommand, return: , log id: aaea0bc 2019-11-12 12:09:44,804Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:44,815Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Removed task 'c964f397-f054-4d81-98d6-052ce5a30168' from DataBase 2019-11-12 12:09:44,887Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] transaction rolled back 2019-11-12 12:09:44,887Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:44,887Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:44,888Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:44,888Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:44,903Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:44,924Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries but max number of retries have been attempted. The command will be marked as Failed. 2019-11-12 12:09:44,925Z INFO [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Exception in invoking callback of command AddVm (97d4103b-acef-4b55-a379-71a1baa04a4f): NullPointerException: 2019-11-12 12:09:44,925Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Error invoking callback method 'onFailed' for 'FAILED' command '97d4103b-acef-4b55-a379-71a1baa04a4f' 2019-11-12 12:09:44,925Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Exception: javax.ejb.EJBTransactionRolledbackException at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:203) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:420) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.1.1.Final.jar:3.1.1.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view2.endAction(Unknown Source) [bll.jar:] at sun.reflect.GeneratedMethodAccessor1071.invoke(Unknown Source) [:1.8.0_222] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.endAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.VmCommand.endActionOnDisks(VmCommand.java:266) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.endWithFailure(AddVmCommand.java:1452) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:736) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:658) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2003) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:520) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:134) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:174) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:67) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:166) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_222] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_222] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] Caused by: java.lang.NullPointerException Please take a look. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo On 10/10/2019, 14:23, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi, I could not check as in the meantime I destroyed my staging setup. Re-provisioning it now and will retest with ansible. On 10/10/2019, 10:51, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi Ondra, That is true, But why does it work from Portal? Image ID is not differ when run when ansible against same template? I will check anyway. On 10/10/2019, 09:47, "Ondra Machacek" <omachace(a)redhat.com<mailto:omachace@redhat.com>> wrote: The error message is: Image does not exist in domain: u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' Can you please check if the image with id 'd680a46a-7188-45f6-b6a7-3333f830c0b4' exists on storage domain with id '935de3ac-a735-4fc8-9161-26bfd751ffc7'? On 08/10/2019 13:05, Vrgotic, Marko wrote: > Dear oVirt, > > Just recently, I have upgraded oVirt staging environment from 4.3.4 to > 4.3.6 release. > > Since then I noticed the issue when creating VMs from template, using > Ansible. When creating VM from template using UI, all works well. > > Tests were executed with same user, with SuperUser priviledges. > > pip freeze > > *ansible==2.7.13* > > asn1crypto==1.0.0 > > bcrypt==3.1.7 > > cffi==1.12.3 > > cryptography==2.7 > > dnspython==1.16.0 > > ipaddress==1.0.22 > > Jinja2==2.10.1 > > lxml==4.4.1 > > MarkupSafe==1.1.1 > > netaddr==0.7.19 > > *ovirt-engine-sdk-python==4.3.3* > > paramiko==2.6.0 > > pycparser==2.19 > > pycurl==7.43.0.3 > > PyNaCl==1.3.0 > > PyYAML==5.1.2 > > six==1.12.0 > > *Logs from engine: * > > 2019-10-08 09:57:42,294Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] Lock Acquired > to object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:57:42,299Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] FINISH, > HSMClearTaskVDSCommand, return: , log id: be932a0 > > 2019-10-08 09:57:42,299Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] FINISH, > SPMClearTaskVDSCommand, return: , log id: 3e25883 > > 2019-10-08 09:57:42,303Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] > BaseAsyncTask::removeTaskFromDB: Removed task > '7e1d68a2-bb49-4456-b228-db851fe6603c' from DataBase > > 2019-10-08 09:57:42,303Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > '4f8a6897-66e8-453b-b706-246cb8505b3e' > > 2019-10-08 09:57:42,375Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] Running > command: AddVmCommand internal: false. Entities affected : ID: > 2e428504-c339-11e9-87ef-00163e3ec101 Type: ClusterAction group CREATE_VM > with role type USER, ID: 9ac6f4ad-58d0-4a7e-b424-91f2d76abcac Type: > VmTemplateAction group CREATE_VM with role type USER, ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: StorageAction group > CREATE_DISK with role type USER > > 2019-10-08 09:57:42,488Z INFO > [org.ovirt.engine.core.bll.AddRngDeviceCommand] (default task-23) > [332c57d4] Running command: AddRngDeviceCommand internal: true. Entities > affected : ID: d5719cf0-2a64-4a28-bbf2-edc1bb05b777 Type: VMAction > group EDIT_VM_PROPERTIES with role type USER > > 2019-10-08 09:57:42,498Z INFO > [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default > task-23) [332c57d4] START, SetVmStatusVDSCommand( > SetVmStatusVDSCommandParameters:{vmId='d5719cf0-2a64-4a28-bbf2-edc1bb05b777', > status='ImageLocked', exitStatus='Normal'}), log id: 2fe004e2 > > 2019-10-08 09:57:42,507Z INFO > [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default > task-23) [332c57d4] FINISH, SetVmStatusVDSCommand, return: , log id: > 2fe004e2 > > 2019-10-08 09:57:42,528Z INFO > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] Running > command: CreateSnapshotFromTemplateCommand internal: true. Entities > affected : ID: 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:42,554Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] START, > CreateVolumeVDSCommand( > CreateVolumeVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', > imageSizeInBytes='8589934592', volumeFormat='COW', > newImageId='9a96018c-ce7d-4c77-a91a-30fc16c7b2ba', imageType='Sparse', > newImageDescription='', imageInitialSizeInBytes='0', > imageId='f9e0c4d9-81dd-4e44-8a71-33e55d4399c6', > sourceImageGroupId='a7e42574-be60-4c94-94a5-cc4b30fdb16f'}), log id: > 37b29c7d > > 2019-10-08 09:57:42,639Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] FINISH, > CreateVolumeVDSCommand, return: 9a96018c-ce7d-4c77-a91a-30fc16c7b2ba, > log id: 37b29c7d > > 2019-10-08 09:57:42,645Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] CommandAsyncTask::Adding > CommandMultiAsyncTasks object for command > 'f1665b66-2032-4a60-944e-c04beace088e' > > 2019-10-08 09:57:42,645Z INFO > [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandMultiAsyncTasks::attachTask: Attaching task > '99abc310-b44b-435c-8d43-922b782a69dc' to command > 'f1665b66-2032-4a60-944e-c04beace088e'. > > 2019-10-08 09:57:42,660Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Adding task > '99abc310-b44b-435c-8d43-922b782a69dc' (Parent Command > 'CreateSnapshotFromTemplate', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling > hasn't started yet.. > > 2019-10-08 09:57:42,934Z INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] EVENT_ID: > USER_ADD_VM_STARTED(37), VM centos-testvm-024 creation was initiated by > mteam@internal-authz. > > 2019-10-08 09:57:42,935Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] BaseAsyncTask::startPollingTask: > Starting to poll task '99abc310-b44b-435c-8d43-922b782a69dc'. > > 2019-10-08 09:57:44,384Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-74) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command 'AddVm' (id: > '1b65f576-365d-4e8b-a920-621516cbdaed') waiting on child command id: > 'f1665b66-2032-4a60-944e-c04beace088e' type:'CreateSnapshotFromTemplate' > to complete > > 2019-10-08 09:57:48,393Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command 'AddVm' (id: > '1b65f576-365d-4e8b-a920-621516cbdaed') waiting on child command id: > 'f1665b66-2032-4a60-944e-c04beace088e' type:'CreateSnapshotFromTemplate' > to complete > > 2019-10-08 09:57:48,394Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: '937776b8-b94d-4178-b9c2-e39fe42cb06c' child commands > '[63409b0b-749a-45d5-85a5-baf55d5fa2a4]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:48,394Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: '937776b8-b94d-4178-b9c2-e39fe42cb06c' Updating status to > 'SUCCEEDED', The command end method logic will be executed by one of its > parent commands. > > 2019-10-08 09:57:48,398Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: 'fdc5571e-a925-4400-a54a-a1ef1dcd23ae' child commands > '[54d68cae-6585-4351-aaf7-fe687905acf4]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:48,398Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: 'fdc5571e-a925-4400-a54a-a1ef1dcd23ae' Updating status to > 'SUCCEEDED', The command end method logic will be executed by one of its > parent commands. > > 2019-10-08 09:57:48,401Z INFO > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'ProcessOvfUpdateForStorageDomain' (id: > 'bdcdae3a-00f6-4176-9d27-0bb0f04912a3') waiting on child command id: > 'b4050edd-62a7-4b83-a4ab-a21c83bf3ade' > type:'CreateOvfVolumeForStorageDomain' to complete > > 2019-10-08 09:57:48,402Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'CreateOvfVolumeForStorageDomain' id: > 'b4050edd-62a7-4b83-a4ab-a21c83bf3ade' child commands > '[937776b8-b94d-4178-b9c2-e39fe42cb06c]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:48,404Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'CreateOvfVolumeForStorageDomain' id: > '5fd47910-95b2-4387-8e26-ea2c80b078a0' child commands > '[fdc5571e-a925-4400-a54a-a1ef1dcd23ae]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:49,416Z INFO > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [390b4c75] Command > 'ProcessOvfUpdateForStorageDomain' id: > '849bd16c-00d2-48a2-8fde-631af0dfe1b5' child commands > '[a80a3657-c813-42c1-a43d-27442b5dc238, > 4f8a6897-66e8-453b-b706-246cb8505b3e]' executions were completed, status > 'SUCCEEDED' > > 2019-10-08 09:57:49,418Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:49,420Z INFO > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' > successfully. > > 2019-10-08 09:57:49,426Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-3) > [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' successfully. > > 2019-10-08 09:57:49,427Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetImageInfoVDSCommand( > GetImageInfoVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 6c02a73c > > 2019-10-08 09:57:49,429Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetVolumeInfoVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > GetVolumeInfoVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 2e131d9d > > 2019-10-08 09:57:49,456Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetVolumeInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@26992dd0, log > id: 2e131d9d > > 2019-10-08 09:57:49,456Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetImageInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@26992dd0, log > id: 6c02a73c > > 2019-10-08 09:57:49,464Z WARN > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] VM is null - no > unlocking > > 2019-10-08 09:57:49,481Z INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] EVENT_ID: > USER_ADD_DISK_FINISHED_SUCCESS(2,021), The disk 'OVF_STORE' was > successfully added. > > 2019-10-08 09:57:49,490Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:49,491Z INFO > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' > successfully. > > 2019-10-08 09:57:49,495Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-3) > [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' successfully. > > 2019-10-08 09:57:49,497Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetImageInfoVDSCommand( > GetImageInfoVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 7a3dad31 > > 2019-10-08 09:57:49,498Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetVolumeInfoVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > GetVolumeInfoVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 5b85c4a5 > > 2019-10-08 09:57:49,522Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetVolumeInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@13555e4, > log id: 5b85c4a5 > > 2019-10-08 09:57:49,522Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetImageInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@13555e4, > log id: 7a3dad31 > > 2019-10-08 09:57:49,529Z WARN > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] VM is null - no > unlocking > > 2019-10-08 09:57:49,540Z INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] EVENT_ID: > USER_ADD_DISK_FINISHED_SUCCESS(2,021), The disk 'OVF_STORE' was > successfully added. > > 2019-10-08 09:57:50,552Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [390b4c75] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:50,575Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 7e77a9a8 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=ee208fd3-e05c-4ba8-aba9-fa521d69a870 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=484c1eed-b03a-4fa0-a12d-0087294dec86 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":false,"Last Updated":null,"Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:50,606Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 7e77a9a8 > > 2019-10-08 09:57:50,614Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock > Acquired to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:50,623Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Running > command: UploadStreamCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:50,624Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > UploadStreamVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > UploadStreamVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c'}), > log id: 531a4a4a > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeVdsBrokerCommand, parameters: > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGUID=ee208fd3-e05c-4ba8-aba9-fa521d69a870 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=484c1eed-b03a-4fa0-a12d-0087294dec86 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ size=30720 > > 2019-10-08 09:57:51,331Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > UploadStreamVDSCommand, return: , log id: 531a4a4a > > 2019-10-08 09:57:51,342Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandAsyncTask::Adding CommandMultiAsyncTasks object for command > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e' > > 2019-10-08 09:57:51,342Z INFO > [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandMultiAsyncTasks::attachTask: Attaching task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' to command > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e'. > > 2019-10-08 09:57:51,351Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Adding > task 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' (Parent Command > 'UploadStream', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling > hasn't started yet.. > > 2019-10-08 09:57:51,355Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > BaseAsyncTask::startPollingTask: Starting to poll task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d'. > > 2019-10-08 09:57:51,358Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock freed > to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 238169f6 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=ee208fd3-e05c-4ba8-aba9-fa521d69a870 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=484c1eed-b03a-4fa0-a12d-0087294dec86 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":true,"Size":30720,"Last Updated":"Tue Oct 08 > 09:57:50 UTC 2019","Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:51,389Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 238169f6 > > 2019-10-08 09:57:51,596Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 59bde70f > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=66984468-bbfa-4941-8c6b-df72a6d920f2 > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=7db55211-041a-40c8-85a6-a2ff1bd182ce > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":false,"Last Updated":null,"Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:51,664Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 59bde70f > > 2019-10-08 09:57:51,672Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock > Acquired to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:51,683Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Running > command: UploadStreamCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > UploadStreamVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > UploadStreamVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c'}), > log id: 1cb812b6 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeVdsBrokerCommand, parameters: > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGUID=66984468-bbfa-4941-8c6b-df72a6d920f2 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=7db55211-041a-40c8-85a6-a2ff1bd182ce > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ size=30720 > > 2019-10-08 09:57:51,933Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > UploadStreamVDSCommand, return: , log id: 1cb812b6 > > 2019-10-08 09:57:51,943Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandAsyncTask::Adding CommandMultiAsyncTasks object for command > 'd0c07988-0e43-4e6d-b86c-e736b28d4537' > > 2019-10-08 09:57:51,943Z INFO > [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandMultiAsyncTasks::attachTask: Attaching task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' to command > 'd0c07988-0e43-4e6d-b86c-e736b28d4537'. > > 2019-10-08 09:57:51,952Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Adding > task 'ba88edd0-8188-472e-8f59-aa3f90e63181' (Parent Command > 'UploadStream', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling > hasn't started yet.. > > 2019-10-08 09:57:51,958Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > BaseAsyncTask::startPollingTask: Starting to poll task > 'ba88edd0-8188-472e-8f59-aa3f90e63181'. > > 2019-10-08 09:57:51,961Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock freed > to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 213b3075 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=66984468-bbfa-4941-8c6b-df72a6d920f2 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=7db55211-041a-40c8-85a6-a2ff1bd182ce > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":true,"Size":30720,"Last Updated":"Tue Oct 08 > 09:57:50 UTC 2019","Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:52,028Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 213b3075 > > 2019-10-08 09:57:52,224Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] Polling and > updating Async Tasks: 7 tasks, 3 tasks to poll now > > 2019-10-08 09:57:52,228Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] Failed in > 'HSMGetAllTasksStatusesVDS' method > > 2019-10-08 09:57:52,235Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] EVENT_ID: > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv> > command HSMGetAllTasksStatusesVDS failed: Invalid parameter: 'size=16777216' > > 2019-10-08 09:57:52,235Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > SPMAsyncTask::PollTask: Polling task > '99abc310-b44b-435c-8d43-922b782a69dc' (Parent Command > 'CreateSnapshotFromTemplate', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned > status 'finished', result 'cleanSuccess'. > > 2019-10-08 09:57:52,240Z ERROR > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > BaseAsyncTask::logEndTaskFailure: Task > '99abc310-b44b-435c-8d43-922b782a69dc' (Parent Command > 'CreateSnapshotFromTemplate', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended > with failure: > > -- Result: 'cleanSuccess' > > -- Message: 'VDSGenericException: VDSErrorException: Failed to > HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'size=16777216', > code = 100', > > -- Exception: 'VDSGenericException: VDSErrorException: Failed to > HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'size=16777216', > code = 100' > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endActionIfNecessary: All tasks of command > 'f1665b66-2032-4a60-944e-c04beace088e' has ended -> executing 'endAction' > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: > 'f1665b66-2032-4a60-944e-c04beace088e'): calling endAction '. > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > SPMAsyncTask::PollTask: Polling task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned > status 'finished', result 'success'. > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > BaseAsyncTask::onTaskEndSuccess: Task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended > successfully. > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) [] > CommandAsyncTask::endCommandAction [within thread] context: Attempting > to endAction 'CreateSnapshotFromTemplate', > > 2019-10-08 09:57:52,242Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endActionIfNecessary: All tasks of command > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e' has ended -> executing 'endAction' > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e'): calling endAction '. > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > SPMAsyncTask::PollTask: Polling task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned > status 'finished', result 'success'. > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > BaseAsyncTask::onTaskEndSuccess: Task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended > successfully. > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [] > CommandAsyncTask::endCommandAction [within thread] context: Attempting > to endAction 'UploadStream', > > 2019-10-08 09:57:52,244Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endActionIfNecessary: All tasks of command > 'd0c07988-0e43-4e6d-b86c-e736b28d4537' has ended -> executing 'endAction' > > 2019-10-08 09:57:52,244Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: > 'd0c07988-0e43-4e6d-b86c-e736b28d4537'): calling endAction '. > > 2019-10-08 09:57:52,245Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [] > CommandAsyncTask::endCommandAction [within thread] context: Attempting > to endAction 'UploadStream', > > 2019-10-08 09:57:52,245Z INFO > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command > [id=f1665b66-2032-4a60-944e-c04beace088e]: Updating status to 'FAILED', > The command end method logic will be executed by one of its parent commands. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'CreateSnapshotFromTemplate' completed, handling the result. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'CreateSnapshotFromTemplate' succeeded, clearing tasks. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] SPMAsyncTask::ClearAsyncTask: > Attempting to clear task '99abc310-b44b-435c-8d43-922b782a69dc' > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' > successfully. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' > successfully. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] START, SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > taskId='99abc310-b44b-435c-8d43-922b782a69dc'}), log id: 66a4e67 > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] START, > HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > HSMTaskGuidBaseVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > taskId='99abc310-b44b-435c-8d43-922b782a69dc'}), log id: 7ec6d8b8 > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' completed, handling the result. > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' succeeded, clearing tasks. > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > taskId='e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d'}), log id: 356927d > > 2019-10-08 09:57:52,250Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' completed, handling the result. > > 2019-10-08 09:57:52,250Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' succeeded, clearing tasks. > > 2019-10-08 09:57:52,250Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' > > 2019-10-08 09:57:52,251Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > taskId='ba88edd0-8188-472e-8f59-aa3f90e63181'}), log id: 75844e34 > > 2019-10-08 09:57:52,259Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] FINISH, HSMClearTaskVDSCommand, > return: , log id: 7ec6d8b8 > > 2019-10-08 09:57:52,259Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] FINISH, SPMClearTaskVDSCommand, > return: , log id: 66a4e67 > > 2019-10-08 09:57:52,260Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] START, > HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > HSMTaskGuidBaseVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > taskId='e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d'}), log id: 3b239769 > > 2019-10-08 09:57:52,263Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] BaseAsyncTask::removeTaskFromDB: > Removed task '99abc310-b44b-435c-8d43-922b782a69dc' from DataBase > > 2019-10-08 09:57:52,263Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > 'f1665b66-2032-4a60-944e-c04beace088e' > > 2019-10-08 09:57:52,297Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] FINISH, > HSMClearTaskVDSCommand, return: , log id: 3b239769 > > 2019-10-08 09:57:52,297Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] FINISH, > SPMClearTaskVDSCommand, return: , log id: 356927d > > 2019-10-08 09:57:52,298Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] START, > HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv<http://ovirt-staging-hv-01.avinity.tv>, > HSMTaskGuidBaseVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > taskId='ba88edd0-8188-472e-8f59-aa3f90e63181'}), log id: 32d44e1 > > 2019-10-08 09:57:52,301Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > BaseAsyncTask::removeTaskFromDB: Removed task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' from DataBase > > 2019-10-08 09:57:52,301Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e' > > 2019-10-08 09:57:52,314Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] FINISH, > HSMClearTaskVDSCommand, return: , log id: 32d44e1 > > 2019-10-08 09:57:52,314Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] FINISH, > SPMClearTaskVDSCommand, return: , log id: 75844e34 > > 2019-10-08 09:57:52,317Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > BaseAsyncTask::removeTaskFromDB: Removed task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' from DataBase > > 2019-10-08 09:57:52,317Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > 'd0c07988-0e43-4e6d-b86c-e736b28d4537' > > 2019-10-08 09:57:54,039Z INFO > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-44) [5830ba58] Command > 'ProcessOvfUpdateForStorageDomain' id: > 'bdcdae3a-00f6-4176-9d27-0bb0f04912a3' child commands > '[b4050edd-62a7-4b83-a4ab-a21c83bf3ade, > 5fd47910-95b2-4387-8e26-ea2c80b078a0, > c11f4d3c-f85b-43c5-986f-a517aed66d7e, > d0c07988-0e43-4e6d-b86c-e736b28d4537]' executions were completed, status > 'SUCCEEDED' > > 2019-10-08 09:57:55,046Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-40) [5830ba58] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:58,077Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-76) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command 'AddVm' id: > '1b65f576-365d-4e8b-a920-621516cbdaed' child commands > '[f1665b66-2032-4a60-944e-c04beace088e]' executions were completed, > status 'FAILED' > > 2019-10-08 09:57:59,094Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:57:59,099Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:57:59,104Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:59,126Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 69861a78 > > 2019-10-08 09:57:59,139Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:57:59,139Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:57:59,139Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 69861a78 > > 2019-10-08 09:57:59,139Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:57:59,149Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Removed > task 'b4c4c723-4ba1-4d5a-86e6-24658a293a04' from DataBase > > 2019-10-08 09:57:59,220Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] transaction > rolled back > > 2019-10-08 09:57:59,220Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:57:59,233Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:57:59,233Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:01,252Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:01,257Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:01,270Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:01,308Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 35fec55 > > 2019-10-08 09:58:01,324Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:01,324Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:01,324Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 35fec55 > > 2019-10-08 09:58:01,324Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:01,335Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Removed > task '6ed0a844-5363-4afd-bffc-a80b6ace69ea' from DataBase > > 2019-10-08 09:58:01,395Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] transaction > rolled back > > 2019-10-08 09:58:01,395Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:01,395Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:01,395Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:01,395Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:01,407Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:01,407Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:05,427Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:05,430Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:05,437Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:05,452Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 6ac726b6 > > 2019-10-08 09:58:05,466Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:05,466Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:05,466Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 6ac726b6 > > 2019-10-08 09:58:05,467Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:05,477Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Removed > task '31bd7a11-203d-4459-ac54-3c398c85771d' from DataBase > > 2019-10-08 09:58:05,490Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] transaction > rolled back > > 2019-10-08 09:58:05,490Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:05,490Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:05,490Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:05,490Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:05,503Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:05,503Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:13,523Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:13,527Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:13,533Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:13,548Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 33164673 > > 2019-10-08 09:58:13,560Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:13,560Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:13,560Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 33164673 > > 2019-10-08 09:58:13,560Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:13,570Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Removed > task 'ce6882de-2d89-4298-bdcf-86d621a54084' from DataBase > > 2019-10-08 09:58:13,583Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] transaction > rolled back > > 2019-10-08 09:58:13,583Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:13,583Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:13,583Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:13,583Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:13,595Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:13,595Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:23,616Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:23,620Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:23,627Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:23,642Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 7346a5bd > > 2019-10-08 09:58:23,663Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:23,663Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:23,663Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 7346a5bd > > 2019-10-08 09:58:23,663Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:23,682Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Removed > task '8483357d-6d4f-418e-9751-70fcb69685cb' from DataBase > > 2019-10-08 09:58:23,695Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] transaction > rolled back > > 2019-10-08 09:58:23,695Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:23,695Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:23,695Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:23,695Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:23,706Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:23,706Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:33,725Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:33,729Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:33,736Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:33,752Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 6d120001 > > 2019-10-08 09:58:33,767Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:33,767Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:33,767Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 6d120001 > > 2019-10-08 09:58:33,767Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:33,778Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Removed > task 'bfb5e88a-0e62-4e03-b02b-a5a0b36f84db' from DataBase > > 2019-10-08 09:58:33,791Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] transaction > rolled back > > 2019-10-08 09:58:33,791Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:33,791Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:33,791Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:33,791Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:33,804Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:33,824Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Failed invoking callback end > method 'onFailed' for command '1b65f576-365d-4e8b-a920-621516cbdaed' > with exception 'null', the callback is marked for end method retries but > max number of retries have been attempted. The command will be marked as > Failed. > > 2019-10-08 09:58:33,824Z INFO > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Exception in invoking callback of > command AddVm (1b65f576-365d-4e8b-a920-621516cbdaed): NullPointerException: > > 2019-10-08 09:58:33,824Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Error invoking callback method > 'onFailed' for 'FAILED' command '1b65f576-365d-4e8b-a920-621516cbdaed' > > 2019-10-08 09:58:33,824Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Exception: > javax.ejb.EJBTransactionRolledbackException > > at > org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:203) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:420) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) > > at > org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) > [weld-ejb-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) > > at > org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) > > at > org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) > > at > org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) > > at > org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) > > at > org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) > > at > org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.endAction(Unknown > Source) [bll.jar:] > > at > sun.reflect.GeneratedMethodAccessor1032.invoke(Unknown Source) [:1.8.0_222] > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_222] > > at java.lang.reflect.Method.invoke(Method.java:498) > [rt.jar:1.8.0_222] > > at > org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) > [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) > [weld-ejb-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) > [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) > [weld-ejb-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) > [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] > > at > org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.endAction(Unknown > Source) [bll.jar:] > > at > org.ovirt.engine.core.bll.VmCommand.endActionOnDisks(VmCommand.java:266) > [bll.jar:] > > at > org.ovirt.engine.core.bll.AddVmCommand.endWithFailure(AddVmCommand.java:1452) > [bll.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:736) > [bll.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:658) > [bll.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2003) > [bll.jar:] > > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) > [utils.jar:] > > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) > [utils.jar:] > > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) > [utils.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:520) > [bll.jar:] > > at > org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:134) > [bll.jar:] > > at > org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:174) > [bll.jar:] > > at > org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:67) > [bll.jar:] > > at > org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:166) > [bll.jar:] > > at > org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) > [bll.jar:] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [rt.jar:1.8.0_222] > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [rt.jar:1.8.0_222] > > at > org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) > [javax.enterprise.concurrent-1.0.jar:] > > at > org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) > [javax.enterprise.concurrent-1.0.jar:] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_222] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [rt.jar:1.8.0_222] > > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222] > > at > org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) > [javax.enterprise.concurrent-1.0.jar:] > > Caused by: java.lang.NullPointerException > > *Ansible output:* > > The full traceback is: > > Traceback (most recent call last): > > File > "/var/folders/40/w2c8fp151854mddz_4n3czwm0000gn/T/ansible_ovirt_vm_payload_uy4lqway/__main__.py", > line 2024, in main > > clone_permissions=module.params['clone_permissions'], > > File > "/var/folders/40/w2c8fp151854mddz_4n3czwm0000gn/T/ansible_ovirt_vm_payload_uy4lqway/ansible_ovirt_vm_payload.zip/ansible/module_utils/ovirt.py", > line 610, in create > > poll_interval=self._module.params['poll_interval'], > > File > "/var/folders/40/w2c8fp151854mddz_4n3czwm0000gn/T/ansible_ovirt_vm_payload_uy4lqway/ansible_ovirt_vm_payload.zip/ansible/module_utils/ovirt.py", > line 341, in wait > > raise Exception("Timeout exceed while waiting on result state of > the entity.") > > Exception: Timeout exceed while waiting on result state of the entity. > > fatal: [centos-testvm-024.avinity.tv<http://centos-testvm-024.avinity.tv> -> localhost]: FAILED! => { > > "changed": false, > > "invocation": { > > "module_args": { > > "affinity_group_mappings": [], > > "affinity_label_mappings": [], > > "allow_partial_import": null, > > "ballooning_enabled": null, > > "boot_devices": null, > > "boot_menu": null, > > "cd_iso": null, > > "clone": false, > > "clone_permissions": false, > > "cloud_init": { > > "authorized_ssh_keys": "ssh-rsa removed pubkey", > > "host_name": "centos-testvm-024.avinity.tv<http://centos-testvm-024.avinity.tv>", > > "user_name": "centos" > > }, > > "cloud_init_nics": [], > > "cloud_init_persist": false, > > "cluster": "avshared1", > > "cluster_mappings": [], > > "comment": "oVirt Template Bakery Validation", > > "cpu_cores": null, > > "cpu_mode": null, > > "cpu_pinning": null, > > "cpu_shares": null, > > "cpu_sockets": null, > > "cpu_threads": null, > > "custom_compatibility_version": null, > > "custom_properties": null, > > "delete_protected": null, > > "description": null, > > "disk_format": "cow", > > "disks": [], > > "domain_mappings": [], > > "fetch_nested": false, > > "force": false, > > "graphical_console": null, > > "high_availability": null, > > "high_availability_priority": null, > > "host": null, > > "host_devices": null, > > "id": null, > > "initrd_path": null, > > "instance_type": "Medium", > > "io_threads": null, > > "kernel_params": null, > > "kernel_path": null, > > "kvm": null, > > "lease": null, > > "lun_mappings": [], > > "memory": null, > > "memory_guaranteed": null, > > "memory_max": null, > > "name": "centos-testvm-024", > > "nested_attributes": [], > > "nics": [ > > { > > "interface": "virtio", > > "name": "nic1", > > "profile_name": "tenant1" > > } > > ], > > "numa_nodes": [], > > "numa_tune_mode": null, > > "operating_system": null, > > "placement_policy": null, > > "poll_interval": 3, > > "quota_id": null, > > "reassign_bad_macs": null, > > "rng_device": null, > > "role_mappings": [], > > "serial_console": null, > > "serial_policy": null, > > "serial_policy_value": null, > > "smartcard_enabled": null, > > "soundcard_enabled": null, > > "sso": null, > > "state": "running", > > "stateless": null, > > "storage_domain": null, > > "sysprep": null, > > "template": "av-centos-75-baseimage_nfs9000", > > "template_version": null, > > "ticket": null, > > "timeout": 180, > > "timezone": null, > > "type": null, > > "usb_support": null, > > "use_latest_template_version": null, > > "vmware": null, > > "vnic_profile_mappings": [], > > "wait": true, > > "watchdog": null, > > "xen": null > > } > > }, > > "msg": "Timeout exceed while waiting on result state of the entity." > > } > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EB5AXLUNU6LRV… > _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ATKD7TYOKB2UH… -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

1 0

VMs Locked - vProtect snapshots
by joseph＠goldman.id.au 25 Nov '19

25 Nov '19

After a few tricky power issues at a site in a less developed country, I had to restart my 3 node ovirt cluster with hosted engine. Everything seemed to come up fine, but that night when my scheduled snapshots/backups etc ran (vProtect) some weird things happened. It created a couple snapshots and couldn't copy them saying disk is locked (even though in the GUI no disks or snapshots were locked) After a couple tries all of a sudden the whole VM is 'locked' this means I can't migrate it, cant start it (if it shuts down) couldn't remove old snapshots etc. I had this happen on another VM and my only solution there was to clone it, but now its happened to all of them and im not sure why. I have run the db utility on command line to unlock everything, nothing is listed as locked, but I still cant run any operations on the VM's. I even cloned them to get them up and running again, but I can't delete the old VM still sitting there (or the disk, snapshots or anything) just keep getting VM is locked. How the hell do I force an unlock of these entities when everything tells me its not locked? I'm almost at a point where I want to back up all VM's and start again but i'd much rather learn some more so I can manage these issues in future. Any help is appreciated! thanks

2 2