- Users - oVirt List Archives

Changing CPU type from Opteron G3 to Epyc with hosted engine in 4.3
by Juhani Rautiainen 06 Feb '19

06 Feb '19

Hi! Now that I have engine and nodes on 4.3 level I'm stuck on upgrading the CPU type. I have 2 node cluster with Epyc processors. It was originally installed with 4.2 so it chose CPU type as Opteron G3 (no Epyc support back then). In Engine 4.3 Epyc is available as CPU type when I choose Compatibility Version: 4.3. Big problem is that it doesn't allow to upgrade CPU because all hosts are not in maintenance: "Error while executing action: Cannot change Cluster CPU type unless all Hosts attached to this Cluster are in Maintenance". Putting all hosts to maintenance is impossible because Engine is hosted in the cluster. I tried with Global HA maintenance, but that didn't help. What is the correct way to solve this problem? Thanks, -Juhani

2 3

Issue when adding new hypervisor to cluster
by Jonas Lindholm 05 Feb '19

05 Feb '19

Background: We had an issue when an operator decided to just install Windows on a running oVirt hypervisor with VMs running. Somehow the Windows installation somehow corrupted some of the storage metadata for one storage domain (The storage domain could not be activated, activation failed after ~5 minutes each time I tried). I was able to move all disks on that storage domain except two to another a new storage domain so all important data was saved. After data was moved I removed the corrupted storage domain from the cluster/datacenter with no issues. I then destroyed the storage domain. The existing hypervisors in the cluster can reboot without any issues with the existing storage domains. So far so good. Issue: However when I try to add an additional hypervisor to the cluster I get an error referencing the corrupted storage domain (UUID is for the bad one that no longer exists). My question is where can the reference still be, in the DB or in any active data structure the hypervisors are using? if it is in the DB I guess some clever SQL statement might be able to remove it but if that's the case I would need help with those SQL statements. Thanks /Jonas *********************************************************** CONFIDENTIALITY AND PRIVACY NOTICE: This e-mail and any attachments are for the exclusive and confidential use of the intended recipient and may constitute non-public information. Personal data in this email is governed by our Privacy Policy at http://business.nasdaq.com/privacy-statement unless explicitly excluded from it; please see the section in the policy entitled “Situations Where This Privacy Policy Does Not Apply” for circumstances where different privacy terms govern emailed personal data. If you received this e-mail in error, disclosing, copying, distributing or taking any action in reliance of this e-mail is strictly prohibited and may be unlawful. Instead, please notify us immediately by return e-mail and promptly delete this message and its attachments from your computer system. We do not waive any work product or other applicable legal privilege(s) by the transmission of this message. ***********************************************************

1 0

rebooting an ovirt cluster
by feral 05 Feb '19

05 Feb '19

How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes? Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues. 1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works. 2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start. This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on. So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine? -- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people.

6 10

Version 4.3.0 installed, but shows 4.2.8
by arturo＠somosespeciales.com 05 Feb '19

05 Feb '19

Hi: I just made a new installation from scratch since the last version of oVirt (ovirt-node-ng-installer-4.3.0-2019020409.el7.iso) and at the end of the installation and entering the cockpit, appears that the version is 4.2.8. Here is a screenshot: https://drive.google.com/uc?export=download&id=1muIsMPeiMYi-FN4Y4NTxSfQ-EeV… Any ideas? Thanks in advance. Greetings

1 2

oVirt 4.3 node-ng host kernel options for nvidia vGPU
by Edward Berger 05 Feb '19

05 Feb '19

Hi, One of our projects wants to try offering VMs with nvidia vGPU. My co-worker had some problems before, so I thought I'd try the latest 4.3 ovirt-node-ng. In the "Edit Host" -> kernel dialog I see two promising checkbox options Hostdev Passthrough & SR-IOV (which adds to kernel line intel_iommu=on) and Blacklist Nouveau (which adds to kernel line rdblacklist=nouveau) but they seem to be acting as mutually exclusive options, when both are selected the kernel command line box is outlined in red and I can't continue on. Am I wrong to want both options?

1 0

Host compatibility issue after upgrade from 4.2.8 to 4.3.0
by ronjero＠gmail.com 05 Feb '19

05 Feb '19

I have a three node (hyper converged) cluster, all hosts are identical hardware wise, however after the upgrade one of the three is getting kicked out of the cluster with the following error: Host is compatible with versions (3.6,4.0,4.1,4.2) and cannot join Cluster... The hosts have SandyBridge processors but do have SSBD: CPU Type: Intel SandyBridge IBRS SSBD Family. virsh -r capabilities | grep ssbd <feature name='ssbd'/> The only other thing I should mention is that I *may* have change the cluster compatibly level to 4.3 while this particular node was in maintenance mode (don't know if that has any influence on this issue). Any help getting this node back into the cluster would be greatly appreciated. Ron.

3 4

ERROR running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode
by mhumaj＠gmail.com 05 Feb '19

05 Feb '19

Hi, We run ovirt upgrade to 4.3, after upgrade we wanted to run engine-setup but we do not know how to put this host which is simply another virtual machine with ovirt-engine. hosted-engine is running on hosts. During execution engine service will be stopped (OK, Cancel) [OK]: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190205121802-l7llrw.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190205121855-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed from the hosted nodes --== Host 2 status ==-- Host ID : 2 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Can anyone please tell me how to put the global maintenance on virtual machine where the ovirt-engine is? not the hosts even if I put them on the global maintenance I am unable to run engine-setup on vm with ovirt-enging. thanks

2 1

[4.3.0] VNC Virt-viewer console not opening
by Nicolas Ecarnot 05 Feb '19

05 Feb '19

Hello, First, congratulations to all of you who worked for this 4.3.0 release, and obviously thank you. Today, I upgraded 4 oVirt setups (4 DC) from 4.2.7 to 4.3.0. I went well on all 4 DCs. But on one of them, when I try to open a console, I see it open as a flash (it opens and closes immediately). I'm using Firefox 64.0 with Ubuntu 18.10, and all my VMs are setup like this : - video type : QXL - Gfx protocol : VNC - VNC Kbd layout : fr and I'm using virt-viewer On the problematic DC, all the VMs are showing the same issue. When I try to use Spice instead of VNc, it is working nicely. When I try to use noVNC, the additional tab opens and shows "Unsupported security types: 19" I tried to track down this issue thanks to the firefox dev console, but it's beyond my understanding. Trying the same with Chromium does the same blinking open/close. I'd rather learn how to provide additionnal debug messages, but /var/log/ovirt-engine/engine.log does not give any useful hint : 2019-02-04 16:57:04,150+01 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-24) [1fb01d42] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 0c3e02b3-7fec-4bb1-b3d6-2e6c228e7278 Type: VMAction group CONNECT_TO_VM with role type USER 2019-02-04 16:57:04,155+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-24) [1fb01d42] START, SetVmTicketVDSCommand(HostName = hv01.prd.sdis38.fr, SetVmTicketVDSCommandParameters:{hostId=' 687c1c01-a5e1-449c-89d2-9713ccfc2487', vmId='0c3e02b3-7fec-4bb1-b3d6-2e6c228e7278', protocol='VNC', ticket='IivrpGHx5zSw', validTime='120', userName='admin', userId='4a340386-851a-11e8-863d-3417ebeef1af', disconnectAction='NONE'} ), log id: 2a897f30 2019-02-04 16:57:04,188+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-24) [1fb01d42] FINISH, SetVmTicketVDSCommand, return: , log id: 2a897f30 2019-02-04 16:57:04,211+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-24) [1fb01d42] EVENT_ID: VM_SET_TICKET(164), User admin@internal-authz initiated console session for VM ad02.ct at.sdis38.fr What could I give to help you help me? -- Nicolas ECARNOT

4 5

Upgrade guide for node from 4.2->4.3
by Juhani Rautiainen 04 Feb '19

04 Feb '19

Hi! Thanks for the new release. I managed to upgrade the engine from 4.2->4.3 with old upgrade instructions but I'm having problems with 4.2 node upgrades. Adding repos with ovirt-release43 doesn't allow node upgrade. Lot's of missing dependencies. Dpdk depencies could be solved with adding Centos Extras, but what repo you should use for openscap, openscap-utils and scap-security-guide packages? -Juhani

3 5

Ovirt cluster unstable; gluster to blame (again)
by Jim Kusznir 04 Feb '19

04 Feb '19

hi all: Once again my production ovirt cluster is collapsing in on itself. My servers are intermittently unavailable or degrading, customers are noticing and calling in. This seems to be yet another gluster failure that I haven't been able to pin down. I posted about this a while ago, but didn't get anywhere (no replies that I found). The problem started out as a glusterfsd process consuming large amounts of ram (up to the point where ram and swap were exhausted and the kernel OOM killer killed off the glusterfsd process). For reasons not clear to me at this time, that resulted in any VMs running on that host and that gluster volume to be paused with I/O error (the glusterfs process is usually unharmed; why it didn't continue I/O with other servers is confusing to me). I have 3 servers and a total of 4 gluster volumes (engine, iso, data, and data-hdd). The first 3 are replica 2+arb; the 4th (data-hdd) is replica 3. The first 3 are backed by an LVM partition (some thin provisioned) on an SSD; the 4th is on a seagate hybrid disk (hdd + some internal flash for acceleration). data-hdd is the only thing on the disk. Servers are Dell R610 with the PERC/6i raid card, with the disks individually passed through to the OS (no raid enabled). The above RAM usage issue came from the data-hdd volume. Yesterday, I cought one of the glusterfsd high ram usage before the OOM-Killer had to run. I was able to migrate the VMs off the machine and for good measure, reboot the entire machine (after taking this opportunity to run the software updates that ovirt said were pending). Upon booting back up, the necessary volume healing began. However, this time, the healing caused all three servers to go to very, very high load averages (I saw just under 200 on one server; typically they've been 40-70) with top reporting IO Wait at 7-20%. Network for this volume is a dedicated gig network. According to bwm-ng, initially the network bandwidth would hit 50MB/s (yes, bytes), but tailed off to mostly in the kB/s for a while. All machines' load averages were still 40+ and gluster volume heal data-hdd info reported 5 items needing healing. Server's were intermittently experiencing IO issues, even on the 3 gluster volumes that appeared largely unaffected. Even the OS activities on the hosts itself (logging in, running commands) would often be very delayed. The ovirt engine was seemingly randomly throwing engine down / engine up / engine failed notifications. Responsiveness on ANY VM was horrific most of the time, with random VMs being inaccessible. I let the gluster heal run overnight. By morning, there were still 5 items needing healing, all three servers were still experiencing high load, and servers were still largely unstable. I've noticed that all of my ovirt outages (and I've had a lot, way more than is acceptable for a production cluster) have come from gluster. I still have 3 VMs who's hard disk images have become corrupted by my last gluster crash that I haven't had time to repair / rebuild yet (I believe this crash was caused by the OOM issue previously mentioned, but I didn't know it at the time). Is gluster really ready for production yet? It seems so unstable to me.... I'm looking at replacing gluster with a dedicated NFS server likely FreeNAS. Any suggestions? What is the "right" way to do production storage on this (3 node cluster)? Can I get this gluster volume stable enough to get my VMs to run reliably again until I can deploy another storage solution? --Jim

11 23