July 2018 - Users - oVirt List Archives

Ovirt cluster unstable; gluster to blame (again)
by Jim Kusznir 04 Feb '19

04 Feb '19

hi all: Once again my production ovirt cluster is collapsing in on itself. My servers are intermittently unavailable or degrading, customers are noticing and calling in. This seems to be yet another gluster failure that I haven't been able to pin down. I posted about this a while ago, but didn't get anywhere (no replies that I found). The problem started out as a glusterfsd process consuming large amounts of ram (up to the point where ram and swap were exhausted and the kernel OOM killer killed off the glusterfsd process). For reasons not clear to me at this time, that resulted in any VMs running on that host and that gluster volume to be paused with I/O error (the glusterfs process is usually unharmed; why it didn't continue I/O with other servers is confusing to me). I have 3 servers and a total of 4 gluster volumes (engine, iso, data, and data-hdd). The first 3 are replica 2+arb; the 4th (data-hdd) is replica 3. The first 3 are backed by an LVM partition (some thin provisioned) on an SSD; the 4th is on a seagate hybrid disk (hdd + some internal flash for acceleration). data-hdd is the only thing on the disk. Servers are Dell R610 with the PERC/6i raid card, with the disks individually passed through to the OS (no raid enabled). The above RAM usage issue came from the data-hdd volume. Yesterday, I cought one of the glusterfsd high ram usage before the OOM-Killer had to run. I was able to migrate the VMs off the machine and for good measure, reboot the entire machine (after taking this opportunity to run the software updates that ovirt said were pending). Upon booting back up, the necessary volume healing began. However, this time, the healing caused all three servers to go to very, very high load averages (I saw just under 200 on one server; typically they've been 40-70) with top reporting IO Wait at 7-20%. Network for this volume is a dedicated gig network. According to bwm-ng, initially the network bandwidth would hit 50MB/s (yes, bytes), but tailed off to mostly in the kB/s for a while. All machines' load averages were still 40+ and gluster volume heal data-hdd info reported 5 items needing healing. Server's were intermittently experiencing IO issues, even on the 3 gluster volumes that appeared largely unaffected. Even the OS activities on the hosts itself (logging in, running commands) would often be very delayed. The ovirt engine was seemingly randomly throwing engine down / engine up / engine failed notifications. Responsiveness on ANY VM was horrific most of the time, with random VMs being inaccessible. I let the gluster heal run overnight. By morning, there were still 5 items needing healing, all three servers were still experiencing high load, and servers were still largely unstable. I've noticed that all of my ovirt outages (and I've had a lot, way more than is acceptable for a production cluster) have come from gluster. I still have 3 VMs who's hard disk images have become corrupted by my last gluster crash that I haven't had time to repair / rebuild yet (I believe this crash was caused by the OOM issue previously mentioned, but I didn't know it at the time). Is gluster really ready for production yet? It seems so unstable to me.... I'm looking at replacing gluster with a dedicated NFS server likely FreeNAS. Any suggestions? What is the "right" way to do production storage on this (3 node cluster)? Can I get this gluster volume stable enough to get my VMs to run reliably again until I can deploy another storage solution? --Jim

11 23

Backup & Restore
by suporte＠logicworks.pt 27 Dec '18

27 Dec '18

------=_Part_20329569_1409874801.1519819859027 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I'm testing backup & restore on Ovirt 4.2. I follow this doc https://www.ovirt.org/documentation/admin-guide/chap-Backups_and_Migration/ Try to restore to a fresh installation but always get this error message: restore-permissions Preparing to restore: - Unpacking file 'back_file' Restoring: - Files Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' Restoring: FATAL: Can't connect to database 'ovirt_engine_history'. Please see '/usr/bin/engine-backup --help'. On the live engine I run # engine-backup --scope=all --mode=backup --file=file_name --log=log_file_name And try to restore on a fresh installation: # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --restore-permissions Any Idea? Thanks -- Jose Ferradeira http://www.logicworks.pt ------=_Part_20329569_1409874801.1519819859027 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: trebuchet ms,sans-serif; font-size: = 12pt; color: #000000"><div data-marker=3D"__QUOTED_TEXT__"><div style=3D"fo= nt-family: Times New Roman; font-size: 10pt; color: #000000;" data-mce-styl= e=3D"font-family: Times New Roman; font-size: 10pt; color: #000000;"><div>H= i,<br></div><br><div><span style=3D"font-size: 11pt; font-family: terminal,= monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;= ">I'm testing backup & restore on Ovirt 4.2.</span><br></div><div><span= style=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style= =3D"font-size: 11pt; font-family: terminal, monaco;">I follow this doc http= s://www.ovirt.org/documentation/admin-guide/chap-Backups_and_Migration/</sp= an><br></div><div><span style=3D"font-size: 11pt; font-family: terminal, mo= naco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;">T= ry to restore to a fresh installation but always get this error message:</s= pan><br></div><br><div><span style=3D"font-size: 11pt; font-family: termina= l, monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, monac= o;">restore-permissions</span><br><span style=3D"font-size: 11pt; font-fami= ly: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-family: term= inal, monaco;">Preparing to restore:</span><br><span style=3D"font-size: 11= pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font= -family: terminal, monaco;">- Unpacking file 'back_file'</span><br><span st= yle=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"f= ont-size: 11pt; font-family: terminal, monaco;">Restoring:</span><br><span = style=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D= "font-size: 11pt; font-family: terminal, monaco;">- Files</span><br><span s= tyle=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"= font-size: 11pt; font-family: terminal, monaco;">Provisioning PostgreSQL us= ers/databases:</span><br><span style=3D"font-size: 11pt; font-family: termi= nal, monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, mon= aco;">- user 'engine', database 'engine'</span><br><span style=3D"font-size= : 11pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; = font-family: terminal, monaco;">Restoring:</span><br><span style=3D"font-si= ze: 11pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt= ; font-family: terminal, monaco;">FATAL: Can't connect to database 'ovirt_e= ngine_history'. Please see '/usr/bin/engine-backup --help'.</span><br></div= ><br><div><span style=3D"font-size: 11pt; font-family: terminal, monaco;" d= ata-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;">On the li= ve engine I run # engine-backup --scope=3Dall --mode=3Dbackup --file= =3Dfile_name --log=3Dlog_file_name</span></div><div><span style=3D"font-siz= e: 11pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt;= font-family: terminal, monaco;"><br data-mce-bogus=3D"1"></span></div><div= ><span style=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-s= tyle=3D"font-size: 11pt; font-family: terminal, monaco;">And try to restore= on a fresh installation:</span><br></div><div><span style=3D"font-size: 11= pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font= -family: terminal, monaco;"># engine-backup --mode=3Drestore --file=3Dfile_= name --log=3Dlog_file_name --provision-db --restore-permissions</span></div= ><div><span style=3D"font-size: 11pt; font-family: terminal, monaco;" data-= mce-style=3D"font-size: 11pt; font-family: terminal, monaco;"><br data-mce-= bogus=3D"1"></span></div><div><span style=3D"font-size: 11pt; font-family: = terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal= , monaco;">Any Idea?<br data-mce-bogus=3D"1"></span></div><div><span style= =3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"font= -size: 11pt; font-family: terminal, monaco;"><br data-mce-bogus=3D"1"></spa= n></div><div><span style=3D"font-size: 11pt; font-family: terminal, monaco;= " data-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;">Thanks= <br data-mce-bogus=3D"1"></span></div><div><span style=3D"font-size: 11pt; = font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-fam= ily: terminal, monaco;"><br data-mce-bogus=3D"1"></span></div><div><span st= yle=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"f= ont-size: 11pt; font-family: terminal, monaco;"><br data-mce-bogus=3D"1"></= span></div><br><div>-- <br></div><div><hr style=3D"width: 100%; height: 2px= ;" data-mce-style=3D"width: 100%; height: 2px;">Jose Ferradeira<br>http://w= ww.logicworks.pt</div></div><br></div></div></body></html> ------=_Part_20329569_1409874801.1519819859027--

4 16

Out of sync, hosts network config differs from DC
by femi adegoke 02 Dec '18

02 Dec '18

Property: default route, host - true, DC - false I have 4 nics. bond0 = 2 x 10g eno1 = ovirtmgmt eno2 = for vm traffic. eno2 says it's out of sync, hosts network config differs from DC, default route: host - true, DC - false. I have tried "sync all networks" but the message still remains See attached. Where can I look to fix the issue?

3 2

oVirt Node on CentOS 7.5 and AMD EPYC Support
by Tobias Scheinert 30 Nov '18

30 Nov '18

Hi, I am currently building a new virtualization cluster with oVirt, using AMD EPYC processors (AMD EPYC 7351P). At the moment I'm running oVirt Node Version 4.2.3 @ CentOS 7.4.1708. We have the situation that the processor type is recognized as "AMD Opteron G3". With this type of instruction set the VMs are not able to do AES in hardware, this results in poor performance in our case. I found some information that tells me that this problem should be solved with CentOS 7.5 --> <https://access.redhat.com/errata/RHEA-2018:1488> My actual questions: - Are there any further information about the AMD EPYC support? - Any information about an update of the oVirt node to CentOS 7.5? Greeting Tobias

7 10

[OT] transferring kvm raw mage from filesystem to lvm based
by Gianluca Cecchi 02 Oct '18

02 Oct '18

Hello, I have a windows 7 VM that is composed by a raw disk on filesystem on Fedora 28. I would like to transfer this VM to another environment, always based on same version of Fedora 28, but where the storage is configured in virt-manager as lvm based, so similar to what happens in oVirt block based storage domains. I'm trying to figure how to transfer the image. I presume I can create a new LV and then leave some bytes for the LVM header into the created LV and then make a sort of dd with offset? Anyone has any hint? Thanks in advance, Gianluca

3 3

Error installing ovirt node
by Junaid Jadoon 19 Sep '18

19 Sep '18

Dear All, I have Ovirt engine 4.2 and node version is 4.2. After installing node in in ovirt engine when i try to install node it gives following error 14:25:37,410+05 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-19) [52669850] Host installation failed for host 'bd8d007a-be92-4075-bba9-6cbeb890a1e5', 'node_2': Command returned failure code 1 during SSH session 'root(a)192.168.20.20' 2018-02-27 14:25:37,416+05 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [52669850] START, SetVdsStatusVDSCommand(HostName = node_2, SetVdsStatusVDSCommandParameters:{hostId='bd8d007a-be92-4075-bba9-6cbeb890a1e5', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2b138e87 2018-02-27 14:25:37,423+05 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [52669850] FINISH, SetVdsStatusVDSCommand, log id: 2b138e87 2018-02-27 14:25:37,429+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-19) [52669850] EVENT_ID: VDS_INSTALL_FAILED(505), Host node_2 installation failed. Command returned failure code 1 during SSH session 'root(a)192.168.20.20'. 2018-02-27 14:25:37,433+05 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-19) [52669850] Lock freed to object 'EngineLock:{exclusiveLocks='[bd8d007a-be92-4075-bba9-6cbeb890a1e5=VDS]', sharedLocks=''}' I have attached log file for your reference Please help me out.

3 8

Failed to synchronize networks of Provider ovirt-provider-ovn
by Kapetanakis Giannis 14 Sep '18

14 Sep '18

Hi, After upgrading to 4.2.1 I have problems with ovn provider. I'm getting "Failed to synchronize networks of Provider ovirt-provider-ovn." I use custom SSL certificate in apache and I guess this is the reason. I've tried to update ovirt-provider-ovn.conf with [OVIRT] #ovirt-ca-file=/etc/pki/ovirt-engine/ca.pem ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem but still no go Any tips on this? thanks G

4 11

Re: Unable to start VM - Image is not a legal chain
by Yedidyah Bar David 14 Sep '18

14 Sep '18

(Changing subject, adding Freddy) On Tue, Jul 10, 2018 at 8:06 PM, Alex K <rightkicktech(a)gmail.com> wrote: > Hi all, > > I did a routine maintenance today (updating the hosts) to ovirt cluster > (4.2) and I have one VM that was complaining about an invalid snapshot. > After shutdown of VM the VM is not able to start again, giving the error: > > VM Gitlab is down with error. Exit message: Bad volume specification > {'serial': 'b6af2856-a164-484a-afe5-9836bbdd14e8', 'index': 0, 'iface': > 'virtio', 'apparentsize': '51838976', 'specParams': {}, 'cache': 'none', > 'imageID': 'b6af2856-a164-484a-afe5-9836bbdd14e8', 'truesize': > '52011008', 'type': 'disk', 'domainID': '142bbde6-ef9d-4a52-b9da-2de533c1f1bd', > 'reqsize': '0', 'format': 'cow', 'poolID': '00000001-0001-0001-0001-000000000311', > 'device': 'disk', 'path': '/rhev/data-center/00000001- > 0001-0001-0001-000000000311/142bbde6-ef9d-4a52-b9da- > 2de533c1f1bd/images/b6af2856-a164-484a-afe5-9836bbdd14e8/ > f3125f62-c909-472f-919c-844e0b8c156d', 'propagateErrors': 'off', 'name': > 'vda', 'bootOrder': '1', 'volumeID': 'f3125f62-c909-472f-919c-844e0b8c156d', > 'diskType': 'file', 'alias': 'ua-b6af2856-a164-484a-afe5-9836bbdd14e8', > 'discard': False}. > > I see also the following error: > > VDSM command CopyImageVDS failed: Image is not a legal chain: > (u'b6af2856-a164-484a-afe5-9836bbdd14e8',) > This error appears a few more times in the list's archive, all of which seem to be related to rather-old bugs (3.5/3.6 times) or storage problems. I assume you use 4.2. Are you sure the corruption happened only now? Did working with snapshots worked well before the upgrade? > > Seems as a corrupt VM disk? > Seems so to me, but I am not a storage expert. > > The VM had 3 snapshots. I was able to delete one from GUI then am not able > to delete the other two as the task fails. Generally I am not allowed to > clone, export or do sth to the VM. > > > Have you encountered sth similar. Any advice? > The lastest post, from 2016, included a workaround, you might (very carefully!) try that. I suggest to also open a bug and attach all relevant logs (engine, vdsm from all relevant hosts, including SPMs at time of snapshot operations and any other host that ran the VM), and try to give accurate reproduction steps. Best regards, -- Didi

3 4

Install oVirt node without Internet repository
by G, Maghesh Kumar (Nokia - IN/Bangalore) 13 Sep '18

13 Sep '18

--_000_HE1PR0702MB3532EC34365D995765A2B7D783A60HE1PR0702MB3532_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Description of problem: Cannot add host - install fails & Install oVirt node without Internet repos= itory Version: Ovirt-4.2 Host is installed with RHEL 7.4 Actual results: Host KVM02 installation failed. Command returned failure code 1 during SSH = session 'root(a)192.175.2.231<mailto:root@192.175.2.231>' I found that right now on my oVirt Nodes in my test environment does not NO= T connect to the internet... Basically, i need the offline repository to access from oVirt nodes without= Internet access. Please guide us how to proceed Regards, Maghesh --_000_HE1PR0702MB3532EC34365D995765A2B7D783A60HE1PR0702MB3532_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang=3D"EN-IN" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN">Hi<o:p></o:p></span= ></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <pre><span style=3D"font-size:12.0pt;color:black">Description of problem:<o= :p></o:p></span></pre> <pre><span style=3D"font-size:12.0pt;color:black">Cannot add host - install= fails & Install oVirt node without Internet repository<o:p></o:p></spa= n></pre> <pre><span style=3D"font-size:12.0pt;color:black"><o:p> </o:p></span><= /pre> <pre><span style=3D"font-size:12.0pt;color:black">Version: Ovirt-4.2<o:p></= o:p></span></pre> <pre><span style=3D"font-size:12.0pt;color:black">Host is installed with RH= EL 7.4<o:p></o:p></span></pre> <pre><span style=3D"font-size:12.0pt;color:black"><o:p> </o:p></span><= /pre> <pre><span style=3D"font-size:12.0pt;color:black">Actual results:<o:p></o:p= ></span></pre> <pre><span style=3D"font-size:12.0pt;color:black">Host KVM02 installation f= ailed. Command returned failure code 1 during SSH session '<a href=3D"mailt= o:root@192.175.2.231"><span style=3D"color:#6699CC">root(a)192.175.2.231</spa= n></a>'<o:p></o:p></span></pre> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN">I found that right = now on my oVirt Nodes in my test environment <b>does not NOT connect to the internet.</b>.. <o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN">Basically, i need t= he offline repository to access from oVirt nodes without Internet access.<o= :p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN">Please guide us how= to proceed<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN">Regards,<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Co= urier New";color:black;mso-fareast-language:EN-IN">Maghesh<o:p></o:p><= /span></p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </body> </html> --_000_HE1PR0702MB3532EC34365D995765A2B7D783A60HE1PR0702MB3532_--

3 7

Engine Setup Error
by Sakhi Hadebe 05 Sep '18

05 Sep '18

Hi, We are deploying the hosted engine on oVirt-Node-4.2.3.1 using the command "hosted-engine --deploy". After providing answers it runs the ansible script and hit the Error when creating glusterfs storage domain. Attached the screenshot of the ERROR. Please help.

5 17