November 2018 - Users - oVirt List Archives

Re: [ovirt-users] Packet loss
by Doron Fediuck 26 Feb '19

26 Feb '19

----_com.android.email_640187878761650 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SGkgS3lsZSzCoApXZSBtYXkgaGF2ZSBzZWVuIHNvbWV0aGluZyBzaW1pbGFyIGluIHRoZSBwYXN0 IGJ1dCBJIHRoaW5rIHRoZXJlIHdlcmUgdmxhbnMgaW52b2x2ZWQuwqAKSXMgaXQgdGhlIHNhbWUg Zm9yIHlvdT/CoApUb255IC8gRGFuLCBkb2VzIGl0IHJpbmcgYSBiZWxsP8Kg ----_com.android.email_640187878761650 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPjwvaGVhZD48Ym9keSA+PGRpdj5IaSBLeWxlLCZuYnNwOzwv ZGl2PjxkaXY+V2UgbWF5IGhhdmUgc2VlbiBzb21ldGhpbmcgc2ltaWxhciBpbiB0aGUgcGFzdCBi dXQgSSB0aGluayB0aGVyZSB3ZXJlIHZsYW5zIGludm9sdmVkLiZuYnNwOzwvZGl2PjxkaXY+SXMg aXQgdGhlIHNhbWUgZm9yIHlvdT8mbmJzcDs8L2Rpdj48ZGl2PlRvbnkgLyBEYW4sIGRvZXMgaXQg cmluZyBhIGJlbGw/Jm5ic3A7PC9kaXY+PC9ib2R5PjwvaHRtbD4= ----_com.android.email_640187878761650--

2 1

ovirt 4.2.7.1 fails to deploy hosted engine on GlusterFS
by hunter86_bg＠yahoo.com 19 Feb '19

19 Feb '19

Hello Community, I'm trying to deploy a hosted engine on GlusterFS which fails with the following error: [ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Failed to fetch Gluster Volume List]\". HTTP response code is 400."} I have deployed GlusterFS via the HyperConverged Option in Cockpit and the volumes are up and running. [root@ovirt1 ~]# gluster volume status engine Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ovirt1:/gluster_bricks/engine/engine 49152 0 Y 26268 Brick ovirt2:/gluster_bricks/engine/engine 49152 0 Y 24116 Brick glarbiter:/gluster_bricks/engine/engi ne 49152 0 Y 23526 Self-heal Daemon on localhost N/A N/A Y 31229 Self-heal Daemon on ovirt2 N/A N/A Y 27097 Self-heal Daemon on glarbiter N/A N/A Y 25888 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks I'm using the following guide : https://ovirt.org/blog/2018/02/up-and-running-with-ovirt-4-2-and-gluster-st… And on step 4 - Storage - I have defined it as follows: Storage Type: Gluster Storage Connection: ovirt1.localdomain:/gluster_bricks/engine/ Mount Options: backup-volfile-servers=ovirt2.localdomain:glarbiter.localdomain Can someone hint me where is the problem ?

3 15

Using the web-ui VM portal through a proxy failing
by Callum Smith 15 Feb '19

15 Feb '19

Dear oVirt Gurus, Using the oVirt user VM portal seems to not work through the squid proxy setup (configured as per the guide). The page loads and login works fine through the proxy, but the asynchronous requests just hang. I've attached a screenshot, but you can see the "api" endpoint just hanging in a web inspector: "https://proxyfqdn/ovirt-engine/api/" [cid:CA42E493-3AD9-45F8-B4C3-C914F059390C@well.ox.ac.uk] This works fine when not going through the proxy. Is there a way to force noVNC HTML as the console mode through the web-ui, or at least have it as an option if not default? The console seems not to work when logged in with a base 'user role'. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum(a)well.ox.ac.uk<mailto:callum@well.ox.ac.uk>

2 14

sun.security.validator
by suporte＠logicworks.pt 06 Feb '19

06 Feb '19

Hi, I'm running Version 4.2.3.8-1.el7, and after reboot the engine machine no longer could login into administration portal with this error: sun.security.validator.ValidatorException: PKIX path validation faile java.security.cert.CertPathValidatorException: validity check failed I'm using a self signed cert. Any idea? Thanks -- Jose Ferradeira http://www.logicworks.pt

5 5

Ovirt cluster unstable; gluster to blame (again)
by Jim Kusznir 04 Feb '19

04 Feb '19

hi all: Once again my production ovirt cluster is collapsing in on itself. My servers are intermittently unavailable or degrading, customers are noticing and calling in. This seems to be yet another gluster failure that I haven't been able to pin down. I posted about this a while ago, but didn't get anywhere (no replies that I found). The problem started out as a glusterfsd process consuming large amounts of ram (up to the point where ram and swap were exhausted and the kernel OOM killer killed off the glusterfsd process). For reasons not clear to me at this time, that resulted in any VMs running on that host and that gluster volume to be paused with I/O error (the glusterfs process is usually unharmed; why it didn't continue I/O with other servers is confusing to me). I have 3 servers and a total of 4 gluster volumes (engine, iso, data, and data-hdd). The first 3 are replica 2+arb; the 4th (data-hdd) is replica 3. The first 3 are backed by an LVM partition (some thin provisioned) on an SSD; the 4th is on a seagate hybrid disk (hdd + some internal flash for acceleration). data-hdd is the only thing on the disk. Servers are Dell R610 with the PERC/6i raid card, with the disks individually passed through to the OS (no raid enabled). The above RAM usage issue came from the data-hdd volume. Yesterday, I cought one of the glusterfsd high ram usage before the OOM-Killer had to run. I was able to migrate the VMs off the machine and for good measure, reboot the entire machine (after taking this opportunity to run the software updates that ovirt said were pending). Upon booting back up, the necessary volume healing began. However, this time, the healing caused all three servers to go to very, very high load averages (I saw just under 200 on one server; typically they've been 40-70) with top reporting IO Wait at 7-20%. Network for this volume is a dedicated gig network. According to bwm-ng, initially the network bandwidth would hit 50MB/s (yes, bytes), but tailed off to mostly in the kB/s for a while. All machines' load averages were still 40+ and gluster volume heal data-hdd info reported 5 items needing healing. Server's were intermittently experiencing IO issues, even on the 3 gluster volumes that appeared largely unaffected. Even the OS activities on the hosts itself (logging in, running commands) would often be very delayed. The ovirt engine was seemingly randomly throwing engine down / engine up / engine failed notifications. Responsiveness on ANY VM was horrific most of the time, with random VMs being inaccessible. I let the gluster heal run overnight. By morning, there were still 5 items needing healing, all three servers were still experiencing high load, and servers were still largely unstable. I've noticed that all of my ovirt outages (and I've had a lot, way more than is acceptable for a production cluster) have come from gluster. I still have 3 VMs who's hard disk images have become corrupted by my last gluster crash that I haven't had time to repair / rebuild yet (I believe this crash was caused by the OOM issue previously mentioned, but I didn't know it at the time). Is gluster really ready for production yet? It seems so unstable to me.... I'm looking at replacing gluster with a dedicated NFS server likely FreeNAS. Any suggestions? What is the "right" way to do production storage on this (3 node cluster)? Can I get this gluster volume stable enough to get my VMs to run reliably again until I can deploy another storage solution? --Jim

11 23

VMs paused - unknown storage error - Stale file handle - distribute 2 - replica 3 volume with sharding
by Marco Lorenzo Crociani 17 Jan '19

17 Jan '19

Hi, I opened a bug on gluster because I have reading errors on files on a gluster volume: https://bugzilla.redhat.com/show_bug.cgi?id=1652548 The files are many of the VMs images of the oVirt DATA storage domain. oVirt pause the vms because unknown storage errors. It's impossibile to copy/clone, manage some snapshots of these vms. The errors on the low level are "stale file handle". Volume is distribute 2 replicate 3 with sharding. Should I open a bug also on oVirt? Gluster 3.12.15-1.el7 oVirt 4.2.6.4-1.el7 Regards, -- Marco Crociani

3 3

Unable to increase memory or CPU on hosted Engine version 4.2.5.2
by Florian Schmid 04 Jan '19

04 Jan '19

Hi, I'm using version 4.2.5.2 and every article I found, I read that with oVirt 4.2, it should be possible to update CPU and memory for hosted Engine in UI. Now, when I'm trying to do this, even being on global maintenance, I always get the same error: Error while executing action: HostedEngine: There was an attempt to change Hosted Engine VM values that are locked. My problem is, that I need to increase the memory, because we had now several times in the last 2 or 3 months an out of memory kill of the java process. What is the best way to increase memory here? Should I do it over the config file? But how is then the OVF updated? BR Florian

2 3

Backup & Restore
by suporte＠logicworks.pt 27 Dec '18

27 Dec '18

------=_Part_20329569_1409874801.1519819859027 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I'm testing backup & restore on Ovirt 4.2. I follow this doc https://www.ovirt.org/documentation/admin-guide/chap-Backups_and_Migration/ Try to restore to a fresh installation but always get this error message: restore-permissions Preparing to restore: - Unpacking file 'back_file' Restoring: - Files Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' Restoring: FATAL: Can't connect to database 'ovirt_engine_history'. Please see '/usr/bin/engine-backup --help'. On the live engine I run # engine-backup --scope=all --mode=backup --file=file_name --log=log_file_name And try to restore on a fresh installation: # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --restore-permissions Any Idea? Thanks -- Jose Ferradeira http://www.logicworks.pt ------=_Part_20329569_1409874801.1519819859027 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: trebuchet ms,sans-serif; font-size: = 12pt; color: #000000"><div data-marker=3D"__QUOTED_TEXT__"><div style=3D"fo= nt-family: Times New Roman; font-size: 10pt; color: #000000;" data-mce-styl= e=3D"font-family: Times New Roman; font-size: 10pt; color: #000000;"><div>H= i,<br></div><br><div><span style=3D"font-size: 11pt; font-family: terminal,= monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;= ">I'm testing backup & restore on Ovirt 4.2.</span><br></div><div><span= style=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style= =3D"font-size: 11pt; font-family: terminal, monaco;">I follow this doc http= s://www.ovirt.org/documentation/admin-guide/chap-Backups_and_Migration/</sp= an><br></div><div><span style=3D"font-size: 11pt; font-family: terminal, mo= naco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;">T= ry to restore to a fresh installation but always get this error message:</s= pan><br></div><br><div><span style=3D"font-size: 11pt; font-family: termina= l, monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, monac= o;">restore-permissions</span><br><span style=3D"font-size: 11pt; font-fami= ly: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-family: term= inal, monaco;">Preparing to restore:</span><br><span style=3D"font-size: 11= pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font= -family: terminal, monaco;">- Unpacking file 'back_file'</span><br><span st= yle=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"f= ont-size: 11pt; font-family: terminal, monaco;">Restoring:</span><br><span = style=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D= "font-size: 11pt; font-family: terminal, monaco;">- Files</span><br><span s= tyle=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"= font-size: 11pt; font-family: terminal, monaco;">Provisioning PostgreSQL us= ers/databases:</span><br><span style=3D"font-size: 11pt; font-family: termi= nal, monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal, mon= aco;">- user 'engine', database 'engine'</span><br><span style=3D"font-size= : 11pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; = font-family: terminal, monaco;">Restoring:</span><br><span style=3D"font-si= ze: 11pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt= ; font-family: terminal, monaco;">FATAL: Can't connect to database 'ovirt_e= ngine_history'. Please see '/usr/bin/engine-backup --help'.</span><br></div= ><br><div><span style=3D"font-size: 11pt; font-family: terminal, monaco;" d= ata-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;">On the li= ve engine I run # engine-backup --scope=3Dall --mode=3Dbackup --file= =3Dfile_name --log=3Dlog_file_name</span></div><div><span style=3D"font-siz= e: 11pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt;= font-family: terminal, monaco;"><br data-mce-bogus=3D"1"></span></div><div= ><span style=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-s= tyle=3D"font-size: 11pt; font-family: terminal, monaco;">And try to restore= on a fresh installation:</span><br></div><div><span style=3D"font-size: 11= pt; font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font= -family: terminal, monaco;"># engine-backup --mode=3Drestore --file=3Dfile_= name --log=3Dlog_file_name --provision-db --restore-permissions</span></div= ><div><span style=3D"font-size: 11pt; font-family: terminal, monaco;" data-= mce-style=3D"font-size: 11pt; font-family: terminal, monaco;"><br data-mce-= bogus=3D"1"></span></div><div><span style=3D"font-size: 11pt; font-family: = terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-family: terminal= , monaco;">Any Idea?<br data-mce-bogus=3D"1"></span></div><div><span style= =3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"font= -size: 11pt; font-family: terminal, monaco;"><br data-mce-bogus=3D"1"></spa= n></div><div><span style=3D"font-size: 11pt; font-family: terminal, monaco;= " data-mce-style=3D"font-size: 11pt; font-family: terminal, monaco;">Thanks= <br data-mce-bogus=3D"1"></span></div><div><span style=3D"font-size: 11pt; = font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-fam= ily: terminal, monaco;"><br data-mce-bogus=3D"1"></span></div><div><span st= yle=3D"font-size: 11pt; font-family: terminal, monaco;" data-mce-style=3D"f= ont-size: 11pt; font-family: terminal, monaco;"><br data-mce-bogus=3D"1"></= span></div><br><div>-- <br></div><div><hr style=3D"width: 100%; height: 2px= ;" data-mce-style=3D"width: 100%; height: 2px;">Jose Ferradeira<br>http://w= ww.logicworks.pt</div></div><br></div></div></body></html> ------=_Part_20329569_1409874801.1519819859027--

4 16

oVirt Node 4.2.7 upgrade fails with broken dependencies ?
by Rogério Ceni Coelho 19 Dec '18

19 Dec '18

Hi all ! Broken dependencies ? [root@nscovirtkvm41prdpoa ~]# yum update Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.ufscar.br * epel: mirror.ci.ifes.edu.br * extras: centos.brnet.net.br * ovirt-4.2: www.gtlib.gatech.edu * ovirt-4.2-epel: mirror.ci.ifes.edu.br * updates: mirror.ufscar.br Resolving Dependencies --> Running transaction check ---> Package collectd.x86_64 0:5.8.0-6.1.el7 will be updated --> Processing Dependency: collectd(x86-64) = 5.8.0-6.1.el7 for package: collectd-disk-5.8.0-6.1.el7.x86_64 --> Processing Dependency: collectd(x86-64) = 5.8.0-6.1.el7 for package: collectd-write_http-5.8.0-6.1.el7.x86_64 ---> Package collectd.x86_64 0:5.8.1-1.el7 will be an update ---> Package collectd-netlink.x86_64 0:5.8.0-6.1.el7 will be updated ---> Package collectd-netlink.x86_64 0:5.8.1-1.el7 will be an update ---> Package collectd-virt.x86_64 0:5.8.0-6.1.el7 will be updated ---> Package collectd-virt.x86_64 0:5.8.1-1.el7 will be an update ---> Package ovirt-hosted-engine-setup.noarch 0:2.2.30-1.el7 will be updated ---> Package ovirt-hosted-engine-setup.noarch 0:2.2.32-1.el7 will be an update --> Finished Dependency Resolution Error: Package: collectd-write_http-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) Requires: collectd(x86-64) = 5.8.0-6.1.el7 Removing: collectd-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-6.1.el7 Updated By: collectd-5.8.1-1.el7.x86_64 (epel) collectd(x86-64) = 5.8.1-1.el7 Available: collectd-5.7.2-1.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-1.el7 Available: collectd-5.7.2-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-3.el7 Available: collectd-5.8.0-2.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-2.el7 Available: collectd-5.8.0-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-3.el7 Available: collectd-5.8.0-5.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-5.el7 Error: Package: collectd-disk-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) Requires: collectd(x86-64) = 5.8.0-6.1.el7 Removing: collectd-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-6.1.el7 Updated By: collectd-5.8.1-1.el7.x86_64 (epel) collectd(x86-64) = 5.8.1-1.el7 Available: collectd-5.7.2-1.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-1.el7 Available: collectd-5.7.2-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-3.el7 Available: collectd-5.8.0-2.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-2.el7 Available: collectd-5.8.0-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-3.el7 Available: collectd-5.8.0-5.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-5.el7 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Cannot upload enabled repos report, is this client registered? [root@nscovirtkvm41prdpoa ~]# yum repolist Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.ufscar.br * epel: mirror.ci.ifes.edu.br * extras: centos.brnet.net.br * ovirt-4.2: www.gtlib.gatech.edu * ovirt-4.2-epel: mirror.ci.ifes.edu.br * updates: mirror.ufscar.br repo id repo name status base/7/x86_64 CentOS-7 - Base 9,911 centos-sclo-rh-release/x86_64 CentOS-7 - SCLo rh 8,099 epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 12,708 extras/7/x86_64 CentOS-7 - Extras 434 ovirt-4.2/7 Latest oVirt 4.2 Release 2,439 ovirt-4.2-centos-gluster312/x86_64 CentOS-7 - Gluster 3.12 262 ovirt-4.2-centos-opstools/x86_64 CentOS-7 - OpsTools - release 666 ovirt-4.2-centos-ovirt42/x86_64 CentOS-7 - oVirt 4.2 582 ovirt-4.2-centos-qemu-ev/x86_64 CentOS-7 - QEMU EV 63 ovirt-4.2-epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 12,708 ovirt-4.2-virtio-win-latest virtio-win builds roughly matching what will be shipped in upcoming RHEL 38 updates/7/x86_64 CentOS-7 - Updates 1,614 zabbix/7/x86_64 Zabbix Official Repository - x86_64 183 zabbix-non-supported/x86_64 Zabbix Official Repository non-supported - x86_64 4 repolist: 49,711 Uploading Enabled Repositories Report Loaded plugins: fastestmirror, product-id, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Cannot upload enabled repos report, is this client registered? [root@nscovirtkvm41prdpoa ~]#

5 6

removal of 3 vdsm hooks
by Dan Kenigsberg 13 Dec '18

13 Dec '18

I've identified 3 ancient vdsm-hooks that have been obsoleted by proper oVirt features. vdsm-hook-isolatedvlan: obsoleted in ovirt-4.2.6 by clean-traffic-gateway filter https://gerrit.ovirt.org/#/q/I396243e1943eca245ab4da64bb286da19f9b47ec vdsm-hook-qos: obsoleted in ovirt-3.3 by https://www.ovirt.org/documentation/sla/network-qos/ vdsm-hook-noipspoof: obsoleted in ovirt-4.0 by choosing the "clean-traffic" filter https://ovirt.org/feature/networkfilter I would like to remove this code from vdsm-4.30, destined for ovirt-4.3. Is there any objection for that? Is anybody still using them? Regards, Dan.

3 3