- Users - oVirt List Archives

Hyperconverged setup ovirt 4.3.x using ansible?
by adrianquintero＠gmail.com 20 Aug '19

20 Aug '19

Hi, I am trying to do a Hyperconverged setup using ansible. So far I have been able to run a playbook to setup gluster but have not been able to identify how to install the Hosted-Engine VM using ansible and tie all the pieces together. Can someone point me in the right direction to deploy a hyperconverged environment using ansible? I have succesfully deployed ovirt 4.3.5 (Hyperconverged) using the web UI. thanks.

3 4

ovirt node install
by Staniforth, Paul 20 Aug '19

20 Aug '19

Hello, on the latest version of the oVirt-node install running nodectl info gives the error Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in <module> CliApplication() File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 200, in CliApplication return cmdmap.command(args) File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 118, in command return self.commands[command](**kwargs) File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 76, in info Info(self.imgbased, self.machine).write() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 46, in __init__ self._fetch_information() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 49, in _fetch_information self._get_bootloader_info() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 62, in _get_bootloader_info bootinfo["entries"][k] = v.__dict__ AttributeError: 'list' object has no attribute '__dict__' Also what is the correct way to update from ovirt node 4.2 to 4.3? I used yum install https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-imag… I then did yum erase ovirt-release42 and rm /etc/yum.repos.d/ovirt-4.2* Regards, Paul S. To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

3 5

numa pinning for high performance
by Vincent Royer 20 Aug '19

20 Aug '19

Is there a good place to learn about cpu pinning and the various settings? I've built a test VM and am trying to understand - I've given it 2 cpus, 4 cores each, 1 thread per core. - The VM is pinned to a specific host with passthrough cpu enabled - numa pinning, first 4 cores on physical socket 1 and the next 4 on physical socket 2 - The host has two sockets each with 20 core cpus I would expect to see a load applied on 8 of the 40 cores on the host during testing. Instead it seems like 32 of 40 cores are working at 100%. What are the effects of this on the performance of the vm? [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png]

3 3

Export as OVA fails
by anthonywest＠alfatron.com.au 20 Aug '19

20 Aug '19

Hi. We are running oVirt 4.3.4 with each of the hosts configured to use local storage. When I attempt to export a large virtual machine as an OVA the process begins but fails about half an hour later. On the oVirt Events page the following entries appear: Starting to export Vm Jessica as a Virtual Appliance Failed to export Vm Jessica as a Virtual Appliance to path /storage/Jessica.ova on Host Trueblood When I log onto the host Trueblood I see that a file called Jessica.ova.tmp has been created. The engine.log file contains the following errors: 2019-08-16 16:40:27,141+10 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Ansible playbook execution failed: Timeout occurred while executing Ansible playbook. 2019-08-16 16:40:27,142+10 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Failed to create OVA. Please check logs for more details: /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20190816161027-trueblood.alfatron.com.au-80ae4c6.log 2019-08-16 16:40:27,161+10 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Failed to create OVA file 2019-08-16 16:40:27,162+10 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Command 'ExportVmToOva' id: '12e98544-67ea-45b0-adb8-c50c8c190ecf' with children [a6c04b63-fe81-4d13-a69f-87b8ac55ceaa] failed when attempting to perform the next operation, marking as 'ACTIVE' 2019-08-16 16:40:27,162+10 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] EngineException: ENGINE (Failed with error ENGINE and code 5001): org.ovirt.engine.core.common.errors.EngineException: EngineException: ENGINE (Failed with error ENGINE and code 5001) at org.ovirt.engine.core.bll.exportimport.ExportOvaCommand.createOva(ExportOvaCommand.java:117) [bll.jar:] at org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand.executeNextOperation(ExportVmToOvaCommand.java:224) [bll.jar:] at org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand.performNextOperation(ExportVmToOvaCommand.java:216) [bll.jar:] at org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:77) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_212] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2019-08-16 16:40:28,304+10 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [3f52073f-b7d2-4238-8ac1-4634678b11b4] Ending command 'org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand' with failure. 2019-08-16 16:40:28,745+10 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [41e553bf] EVENT_ID: IMPORTEXPORT_EXPORT_VM_TO_OVA_FAILED(1,225), Failed to export Vm Jessica as a Virtual Appliance to path /storage/Jessica.ova on Host Trueblood Any suggestions/help on solving this would be much appreciated. Anthony

2 1

Issues with oVirt-Engine start - oVirt 4.3.4
by Vrgotic, Marko 19 Aug '19

19 Aug '19

Dear oVirt, While working on a procedure to get the NFS v4 mount from Netapp, working on oVIrt, following steps came out to be the way to go in regards of setting it up for oVIrt SHE and VM Guests: * mkdir /mnt/rhevstore * mount -t nfs 10.20.30.40:/ovirt_hosted_engine /mnt/rhevstore * chown -R 36.36 /mnt/rhevstore * chmod -R 755 /mnt/rhevstore * umount /mnt/rhevstore This works fine, and it needs to be executed on each Hypervisor, before its provisioned into oVirt. However, just today I have discovered that command chmod -R 755 /mnt/rhevstore, if executed on new to be added Hypervisor, after oVirt is already running, it brings the oVirt Engine into broken state. The moment I executed the above on 3rd Hypervisor, before provisioning it into oVirt, following occurred: * Engine threw following error: * 2019-08-19 13:16:31,425Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-82) [] Failed in 'SpmStatusVDS' method * Connection was lost: * packet_write_wait: Connection to 10.210.11.10 port 22: Broken pipe * And VDSM on SHE Hosting hypervisor started logging errors like: * 2019-08-19 15:00:52,340+0000 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) * 2019-08-19 15:00:53,865+0000 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/2 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f59442c3d50> at 0x7f59442c3b90> timeout=15, duration=225.00 at 0x7f592476df90> task#=578 at 0x7f59442ef910>, traceback: * File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap * self.__bootstrap_inner() * File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner * self.run() * File: "/usr/lib64/python2.7/threading.py", line 765, in run * self.__target(*self.__args, **self.__kwargs) * File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run * ret = func(*args, **kwargs) * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run * self._execute_task() * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task * task() * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ * self._callable() * File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 186, in __call__ * self._func() * File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 481, in __call__ * stats = hostapi.get_stats(self._cif, self._samples.stats()) * File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 79, in get_stats * ret['haStats'] = _getHaInfo() * File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 177, in _getHaInfo * stats = instance.get_all_stats() * File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats * stats = broker.get_stats_from_storage() * File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in get_stats_from_storage * result = self._proxy.get_stats() * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ * return self.__send(self.__name, args) * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request * verbose=self.__verbose * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request * return self.single_request(host, handler, request_body, verbose) * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request * response = h.getresponse(buffering=True) * File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse * response.begin() * File: "/usr/lib64/python2.7/httplib.py", line 444, in begin * version, status, reason = self._read_status() * File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status * line = self.fp.readline(_MAXLINE + 1) * File: "/usr/lib64/python2.7/socket.py", line 476, in readline * data = self._sock.recv(self._rbufsize) (executor:363) * 2019-08-19 15:00:54,103+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) I am unable to boot the Engine VM – it end up in Status ForceStop Hosted-engine –vm-status shows: [root@ovirt-sj-02 ~]# hosted-engine --vm-status The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. But, storage is mounted and reachable, as well as ovirt-ha-agent running: [root@ovirt-sj-02 ~]# systemctl status ovirt-ha-agent ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-08-19 14:57:07 UTC; 23s ago Main PID: 43388 (ovirt-ha-agent) Tasks: 2 CGroup: /system.slice/ovirt-ha-agent.service └─43388 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent Can somebody help me with what to do? — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

1 2

Hosted Engine on seperate L2 network from nodes?
by Dan Poltawski 19 Aug '19

19 Aug '19

For some security requirements, I’ve been asked if it’s possible to segregate the hosted engine from the physical nodes, with specific firewalling for access to do node/ storage operations (I’m using managed block storage). Is this an approach others us, or is it better practice and just ensure the nodes and engine are all sharing the same network? Thanks, dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

2 1

Adding VLANs to a single-host, self-hosted-engine oVirt deployment?
by Derek Atkins 16 Aug '19

16 Aug '19

HI, I've got a single oVirt host running a self-contained hosted-engine deployment. When I set it up I did not use VLANs in my network. I am in the process of moving my equipment, and in part of this move I would like to introduce VLANs into my network infrastructure. The documentation seems to imply that to add virtual networks and/or VLANs to a host that I need to put it into maintenance mode, configure it in the engine, and then resync the network. However, I don't think I can do that with a single-host environment. If I put the host into local maint mode, it will try to offload all my VMs, including the engine, which obviously it cannot do because there is no other host to migrate them to. So what's the approach to add VLANs in this situation? I should add that this system started at 4.0, and I'm still only running 4.1 (although I do plan to upgrade to 4.2 as part of this move). I'm hesitant to upgrade further because of the impending removal of SDK-3 -- I am depending on a script that uses ovirt-shell which I keep being told is going away. If ovirt-shell is still in 4.3 then I might consider upgrading to that as well. :) Thanks. -derek -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

2 3

Can't add a pre-existing (storage-only) gluster member as a compute-also node using self-hosted hyper-convergent engine
by thomas＠hoberg.net 16 Aug '19

16 Aug '19

Intro: Trying to add oVirt self-hosted engine to a pre-existing set of system already using Gluster storage seems to fail, because ‘hosted engine wizard’ seems to create its own peer pool for managed hosts, instead of joining the one given for storage. Context: I am testing on a set of four Atom Goldmont+ boxes, which are silent/low-power, cheap, fast-enough for edge workloads and even take 32GB of RAM these days. But for various, sometimes even good reasons, they are not a mainline platform and I attributed many problems I faced to the niche hardware--sometimes correctly. Because the three-node hyperconverged setup has very exacting requirements, hard to meet in pre-existing machines (and in my case many initial failures with too little insight), I created the storage gluster separately first and then used the “hosted engine” wizard to set up the hosted engine on one of the Atom nodes. I used CentOS7 (fresh install and latest updates) on the primary nodes, not the oVirt node image, because some of my targets are big HPC machines, that are supposed to run oVirt for support services in a small niche, while Nvidia-Docker/SLURM workloads dominate. I assumed that if I were to split the storage and the orchestration setup into distinct steps, it would give me both better insight and flexibility to expand/transform the storage and the compute without losing any of the self-hosted hyperconvergeance comfort. Problem: I had all kinds of problem to get the hosted engine to run all the way through on the Atoms, it typically stopped just shy of the final launch as a VM on the Gluster storage. I eventually stumbled across this message from Yedidyah Bar David: https://lists.ovirt.org/pipermail/users/2018-March/087923.html I then had a look at the engine database and found that indeed the compute nodes were all in a separate gluster newly created by the hosted-engine setup and evidently used for cluster synchronization, instead of joining the pool already used for the bulk of the storage. I don’t know if this would be considered an error that needs fixing, an issue that can be avoided using a manual configuration, or something else. I believe it could use some highlighting in the documentation.

1 2

Hyperconverged on a self-made Gluster: Considered normal or 'way-off'?
by thomas＠hoberg.net 16 Aug '19

16 Aug '19

I use three very low power target nodes (J5005 Intel Atoms, 24GB RAM, 250GB SSD) to experiment with hyperconverged oVirt, and some bigger machines to act as pure compute nodes, preferably CentOS based, potentially oVirt Node image based. I've used both CentOS7 and oVirt Node image bases (all latest versions and patches) and typically I use either a single partition and xfs root file system to avoid partitioning the constricted space on CentOS or the recommended partition layout with the oVirt node images, which still consumes the single primary SSD /dev/sda. In both cases I don't have another device or partition left for the three-node hyperconverged wizard's ansible scripts to play with. And since I want to understand better what is going on under the hood in any case I tried to separate the Gluster setup and the hosted-engine setup. Question: Is the impact of that decision perhaps far bigger than I imagined? So I went with the alternative A Hosted-Engine wizard, where the storage was already provisioned and I had set up 2+1 Gluster storage using bricks right on the xfs, assuming I would achieve something very similar to what the three node wizard would simplify, except without the LVM/VDO layer all done with Ansible. But I am running into all kinds of problems and what continues to irritate me is the note in the manuals, that clusters need to be either 'compute' or 'storage' but cannot be both--yet that's the very point of hyperconvergence... So basically I am wondering whether at this point there is a strict bifurifcation of oVirt between the hyperconverged variant and the 'Hosted-Engine' on existing storage, without any middle ground. Or is it ok to do a two-phased approach, where you separate out the Gluster aspect and the Hosted-Engine *but with physical servers that are doing both storage and compute*? Also, I am wondering about the 'Enable Virt Service' and 'Enable Gluster Service' tick boxes in the Cluster property pages: For my 'hosted engine' wizard generated Cluster 'gluster' was unticked, even if it did run on Gluster storage. And while ticking 'gluster' in addition had no visible effect, unticking either one of the two is now impossible. I could not find any documentation on the significance or effect of these boxes. While the prepared three node replicated storage Gluster for the /engine was running on CentOS nodes, the Hosted-Engine VM was then created on additional oVirt Node hosts using that Gluster storage (trying to do the same on CentOS nodes always fails, separate topic), with the assumption that I could then push the hosted-engine VM back to the gluster storage nodes and get rid of the temporary compute-only host used for hosted engine setup. I then tried to add the Gluster nodes as hosts (and hosted-engine backup nodes), that also failed. Whether it failed because they were CentOS based or because they were already used for storage (and there is this hidden exclusivity), I would like to have some clarity on. So to repeat once again my main question: Is the (one or three) node hyperconverged oVirt setup a completely different thing from the hosted-engine on Gluster? Are you supposed to be able to add extra compute hosts, storage etc. to be managed by a three-node HC based hosted-engine? Are you supposed to be able to expand the three-node HC also in terms of storage, if perhaps currently only in quorum-maintaining multiples? Is oVirt currently actually strictly mandating 100% segregation between compute and storage unless you use the wholly contained single/triple HC setups?

1 0

Disk Allocation policy changes after a snapshot
by Kevin Doyle 16 Aug '19

16 Aug '19

Hi I have linux VM with 2 disks one for OS is sparse the other is for a Database and is Preallocated. When I take a snapshot of the VM both disks change to sparse policy but the disks in the snapshot are 1 sparse and 1 allocated. Before the snapshot the VM was running fine, now it crashes when data is written to the database. When I delete the snapshot the disks go back to 1 sparse and 1 allocated. Has anyone else seen this happen. Ovirt is 4.3.2.1-1.el7 and it is running on a hostedEngine Many thanks Kevin

4 5