February 2022 - Users - oVirt List Archives

Unable to migrate VMs to a newly upgraded Ovirt node host
by Giulio Casella 25 Feb '22

25 Feb '22

Hi guys, I just faced a problem after updating a host. I cannot migrate VM to updated host. Here's the error I see trying to migrate a VM to that host. Dec 16 10:13:11 host01.ovn.di.unimi.it systemd[1]: Starting Network Manager Script Dispatcher Service... Dec 16 10:13:11 host01.ovn.di.unimi.it libvirtd[5667]: Unable to read from monitor: Connection reset by peer Dec 16 10:13:11 host01.ovn.di.unimi.it libvirtd[5667]: internal error: qemu unexpectedly closed the monitor: 2021-12-16T10:13:00.447480Z qemu-kvm: -numa node,nodeid=0,cpus=0-15,mem=8192: warning: Parameter -numa node,mem is deprecated, use -numa node,memdev instead 2021-12-16T10:13:11.158057Z qemu-kvm: Failed to load pckbd:kbd 2021-12-16T10:13:11.158114Z qemu-kvm: error while loading state for instance 0x0 of device 'pckbd' 2021-12-16T10:13:11.158744Z qemu-kvm: load of migration failed: No such file or directory Dec 16 10:13:11 host01.ovn.xx.xxxxx.it kvm[35663]: 0 guests now active Instead I can start VM on that host, and migrate away VM from that host. Rolling back to ovirt-node-ng-4.4.9.1-0.20211207.0+1 via host console restores full functionality. The affected version is ovirt-node-ng-4.4.9.3-0.20211215.0+1 (and also previous one, I don't remember precisely, it was another async release). Any ideas? TIA, gc

3 2

Re: Migrate Hosted-Engine from one NFS mount to another NFS mount
by Sketch 24 Feb '22

24 Feb '22

On Wed, 23 Feb 2022, Matthew.Stier(a)fujitsu.com wrote: > I’ve always been told that migrating self-hosted-engine storage was a > backup, shutdown, and rebuild from backup procedure. > > In my iscsi environment it has never worked. (More due to the history of my > environment, than the procedure itself.) This didn't work for me either, though it may have had to do with the many issues I had moving from 4.3 to 4.4. What did work was backing up and restoring to a standalone engine. I had initially planned to migrate it back to a self-hosted setup later, but found that this setup is a lot less temperamental than the self-hosted engine. It also doesn't have to be a dedicated machine, it can be a VM hosted outside of oVirt, which may also be useful if you're hosting some stuff needed for bootstrapping the environment like DNS, NTP, etc. outside of oVirt.

3 2

Note about oVirt and Let's Encrypt SSL certs
by Chris Adams 24 Feb '22

24 Feb '22

I was setting up a new oVirt cluster yesterday, and deployed a Let's Encrypt SSL cert on it for the website. After that, I noticed that oVirt was getting errors synchronizing networks with ovirt-provider-ovn. It appears that the python library used for SSL by ovirt-provider-ovn has the same issue as older OpenSSL versions, and can't handle the default Let's Encrypt root cert path; the path used for old Android compatibility can end with an expired cert that's still in the CA store (even though there's another verification path that doesn't end with an expired cert). The solution was to switch the Let's Encrypt cert to the "ISRG Root X1" chain (which is fine, since I don't log in to oVirt from Android 7 devices). Just an FYI for anyone else using a Let's Encrypt cert (or other cert with a similar expired root path, they aren't the only one). -- Chris Adams <cma(a)cmadams.net>

1 0

Re: Gluster Performance issues
by Alex Morrison 24 Feb '22

24 Feb '22

Hello Sunil, [root@ovirt1 ~]# gluster --version glusterfs 8.6 same on all hosts On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu(a)redhat.com> wrote: > Hi, > > Which version of gluster is in use? > > Regards, > > Sunil kumar Acharya > > Red Hat > > <https://www.redhat.com> > > T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/> > > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > > > On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex(a)discoverygarden.ca> > wrote: > >> Hello All, >> >> We have 3 servers with a raid 50 array each, we are having extreme >> performance issues with our gluster, writes on gluster seem to take at >> least 3 times longer than on the raid directly. Can this be improved? I've >> read through several other performance issues threads but have been unable >> to make any improvements >> >> "gluster volume info" and "gluster volume profile vmstore info" is below >> >> >> ========================================================================================================================= >> >> -Inside Gluster - test took 35+ hours: >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s >> 600G -n 0 -m TEST -f -b -u root >> Using uid:0, gid:0. >> Writing intelligently...done >> Rewriting...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Version 1.98 ------Sequential Output------ --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> TEST 600G 35.7m 17 5824k 7 112m 13 >> 182.7 6 >> Latency 5466ms 12754ms 3499ms >> 1589ms >> >> >> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, >> >> >> ========================================================================================================================= >> >> -Outside Gluster - test took 18 minutes: >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s >> 600G -n 0 -m TEST -f -b -u root >> Using uid:0, gid:0. >> Writing intelligently...done >> Rewriting...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Version 1.98 ------Sequential Output------ --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> TEST 600G 567m 78 149m 30 307m 37 >> 83.0 57 >> Latency 205ms 4630ms 1450ms >> 679ms >> >> >> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, >> >> >> ========================================================================================================================= >> >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info >> Volume Name: engine >> Type: Replicate >> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine >> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine >> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine >> Options Reconfigured: >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> network.ping-timeout: 30 >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> server.event-threads: 4 >> client.event-threads: 4 >> cluster.choose-local: off >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> transport.address-family: inet >> storage.fips-mode-rchecksum: on >> nfs.disable: on >> performance.client-io-threads: on >> diagnostics.latency-measurement: on >> diagnostics.count-fop-hits: on >> >> Volume Name: vmstore >> Type: Replicate >> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore >> Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore >> Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore >> Options Reconfigured: >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> network.ping-timeout: 20 >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> server.event-threads: 4 >> client.event-threads: 4 >> cluster.choose-local: off >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> transport.address-family: inet >> storage.fips-mode-rchecksum: on >> nfs.disable: on >> performance.client-io-threads: on >> diagnostics.latency-measurement: on >> diagnostics.count-fop-hits: on >> server.tcp-user-timeout: 20 >> server.keepalive-time: 10 >> server.keepalive-interval: 2 >> server.keepalive-count: 5 >> cluster.lookup-optimize: off >> >> >> ========================================================================================================================= >> >> [root@ovirt1 ~]# gluster volume profile vmstore info >> Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore >> --------------------------------------------------------- >> Cumulative Stats: >> Block Size: 1b+ 4b+ >> 8b+ >> No. of Reads: 0 0 >> 0 >> No. of Writes: 4021 2 >> 64 >> >> Block Size: 64b+ 128b+ >> 256b+ >> No. of Reads: 0 134 >> 405 >> No. of Writes: 2 32 >> 119 >> >> Block Size: 512b+ 1024b+ >> 2048b+ >> No. of Reads: 2890 2617 >> 3059 >> No. of Writes: 113881 93551 >> 131776 >> >> Block Size: 4096b+ 8192b+ >> 16384b+ >> No. of Reads: 1096597 164931 >> 181569 >> No. of Writes: 41091001 20499776 >> 3230472 >> >> Block Size: 32768b+ 65536b+ >> 131072b+ >> No. of Reads: 212970 96574 >> 244017 >> No. of Writes: 1304688 991154 >> 3412866 >> >> Block Size: 262144b+ 524288b+ >> 1048576b+ >> No. of Reads: 0 0 >> 0 >> No. of Writes: 15 18 >> 65166 >> >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 6396 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 189759 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 31292 >> RELEASEDIR >> 0.00 159.99 us 44.37 us 278.66 us 79 >> STAT >> 0.00 364.42 us 174.87 us 644.45 us 96 >> LINK >> 0.00 2714.10 us 203.53 us 47397.28 us 26 >> RMDIR >> 0.00 688.26 us 72.39 us 53857.63 us 186 >> SETXATTR >> 0.00 974.01 us 271.02 us 27298.52 us 176 >> RENAME >> 0.00 5254.64 us 159.48 us 84454.46 us 35 >> DISCARD >> 0.00 1133.84 us 86.47 us 87096.92 us 186 >> REMOVEXATTR >> 0.00 7907.47 us 221.82 us 182276.21 us 29 >> FALLOCATE >> 0.00 8304.51 us 311.89 us 42725.81 us 34 >> MKDIR >> 0.00 5758.19 us 296.39 us 21950.99 us 81 >> TRUNCATE >> 0.00 401.16 us 23.68 us 132668.93 us 3621 >> LK >> 0.00 545.48 us 14.22 us 1076517.76 us 6796 >> READDIR >> 0.00 2250.78 us 336.61 us 1071764.61 us 4257 >> CREATE >> 0.00 531.76 us 1.35 us 1723046.94 us 31292 >> OPENDIR >> 0.01 2058.67 us 196.74 us 1587752.27 us 21986 >> MKNOD >> 0.01 1241.28 us 22.61 us 1548642.70 us 52674 >> READDIRP >> 0.01 67508.32 us 100.01 us 3002069.81 us 1100 >> FTRUNCATE >> 0.01 412.74 us 10.94 us 5111334.35 us 209731 >> FLUSH >> 0.02 78668.59 us 183.06 us 3012629.87 us 1222 >> SETATTR >> 0.03 761.13 us 21.13 us 4611854.15 us 249094 >> STATFS >> 0.03 1132.33 us 45.51 us 3159531.87 us 185654 >> OPEN >> 0.03 2010.33 us 14.43 us 3158098.85 us 107824 >> GETXATTR >> 0.04 2765.29 us 61.38 us 3159208.13 us 100484 >> XATTROP >> 0.13 1367.04 us 51.46 us 8450420.87 us 595299 >> FSTAT >> 0.46 157440.90 us 47.62 us 13405712.93 us 18218 >> UNLINK >> 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 >> LOOKUP >> 1.67 16645.99 us 12.80 us 6869488.64 us 631331 >> ENTRYLK >> 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 >> READ >> 4.31 56942.36 us 16.57 us 23654127.65 us 475563 >> INODELK >> 6.09 660.89 us 11.02 us 20571410.32 us 57908331 >> FINODELK >> 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 >> WRITE >> 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 >> FXATTROP >> 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 78458 >> UPCALL >> >> Duration: 404667 seconds >> Data Read: 59850935724 bytes >> Data Written: 1070158035683 bytes >> >> Interval 43 Stats: >> Block Size: 1b+ 256b+ >> 512b+ >> No. of Reads: 0 14 >> 3 >> No. of Writes: 12 4 >> 252 >> >> Block Size: 1024b+ 2048b+ >> 4096b+ >> No. of Reads: 0 0 >> 8105 >> No. of Writes: 100 34 >> 147355 >> >> Block Size: 8192b+ 16384b+ >> 32768b+ >> No. of Reads: 1559 1630 >> 1494 >> No. of Writes: 86013 17318 >> 8687 >> >> Block Size: 65536b+ 131072b+ >> No. of Reads: 532 764 >> No. of Writes: 8617 36317 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 16 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 665 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 52 >> RELEASEDIR >> 0.00 181.75 us 181.75 us 181.75 us 1 >> STAT >> 0.00 914.55 us 858.55 us 970.56 us 2 >> TRUNCATE >> 0.00 792.11 us 681.24 us 1051.32 us 4 >> RENAME >> 0.00 374.44 us 31.86 us 792.10 us 20 >> READDIR >> 0.00 399.48 us 306.97 us 562.69 us 28 >> UNLINK >> 0.00 752.40 us 636.14 us 907.29 us 16 >> CREATE >> 0.00 196.62 us 33.27 us 2586.77 us 336 >> GETXATTR >> 0.00 1107.98 us 288.51 us 29692.53 us 80 >> MKNOD >> 0.00 1074.80 us 74.41 us 2121.03 us 126 >> READDIRP >> 0.00 3119.52 us 3.17 us 157121.44 us 52 >> OPENDIR >> 0.01 972.14 us 21.72 us 329342.21 us 635 >> FLUSH >> 0.03 2122.15 us 31.97 us 377538.16 us 732 >> STATFS >> 0.06 4478.89 us 66.09 us 521289.10 us 649 >> OPEN >> 0.10 10273.00 us 88.19 us 478524.77 us 472 >> XATTROP >> 0.16 6969.92 us 16.14 us 385156.98 us 1138 >> ENTRYLK >> 0.20 4635.40 us 64.29 us 1098319.92 us 2097 >> FSTAT >> 0.65 4098.07 us 46.85 us 919617.66 us 7797 >> LOOKUP >> 2.36 87465.33 us 34.75 us 1224052.07 us 1317 >> INODELK >> 4.37 13897.55 us 57.89 us 1984970.27 us 15336 >> READ >> 5.24 1265.46 us 16.22 us 1638678.32 us 202165 >> FINODELK >> 21.05 3370.28 us 144.44 us 2264130.49 us 304724 >> WRITE >> 31.00 6411.68 us 39.84 us 2190355.42 us 235969 >> FXATTROP >> 34.75 6107.95 us 60.26 us 2270252.44 us 277643 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 368 >> UPCALL >> >> Duration: 1132 seconds >> Data Read: 292367566 bytes >> Data Written: 7495339692 bytes >> >> Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore >> --------------------------------------------------------- >> Cumulative Stats: >> Block Size: 1b+ 4b+ >> 8b+ >> No. of Reads: 0 0 >> 0 >> No. of Writes: 4802 2 >> 64 >> >> Block Size: 64b+ 128b+ >> 256b+ >> No. of Reads: 0 138 >> 828 >> No. of Writes: 2 32 >> 131 >> >> Block Size: 512b+ 1024b+ >> 2048b+ >> No. of Reads: 3595 4590 >> 6161 >> No. of Writes: 128976 99680 >> 132413 >> >> Block Size: 4096b+ 8192b+ >> 16384b+ >> No. of Reads: 930031 128995 >> 150468 >> No. of Writes: 48357551 23957030 >> 3689474 >> >> Block Size: 32768b+ 65536b+ >> 131072b+ >> No. of Reads: 154039 71949 >> 790010 >> No. of Writes: 1438859 1052045 >> 3715427 >> >> Block Size: 262144b+ 524288b+ >> 1048576b+ >> No. of Reads: 9 11 >> 40269 >> No. of Writes: 5 20 >> 50757 >> >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 7629 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 228577 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 41524 >> RELEASEDIR >> 0.00 374.22 us 219.35 us 574.87 us 96 >> LINK >> 0.00 1712.22 us 164.90 us 22337.79 us 41 >> DISCARD >> 0.00 2840.25 us 245.24 us 46839.63 us 26 >> RMDIR >> 0.00 3660.92 us 229.86 us 86252.56 us 35 >> FALLOCATE >> 0.00 876.48 us 297.53 us 16685.63 us 188 >> RENAME >> 0.00 6660.10 us 379.21 us 52962.79 us 34 >> MKDIR >> 0.00 957.48 us 80.36 us 218059.88 us 355 >> REMOVEXATTR >> 0.00 1234.85 us 76.50 us 322955.27 us 355 >> SETXATTR >> 0.00 2326.62 us 84.07 us 250041.25 us 196 >> STAT >> 0.00 9660.45 us 289.50 us 295106.01 us 87 >> TRUNCATE >> 0.00 486.10 us 24.12 us 869966.79 us 5260 >> LK >> 0.00 453.69 us 18.32 us 426653.18 us 8135 >> READDIR >> 0.00 1832.38 us 296.61 us 261299.81 us 5050 >> CREATE >> 0.00 386.77 us 1.31 us 1300836.12 us 41524 >> OPENDIR >> 0.00 712.06 us 29.85 us 1441115.02 us 34874 >> READDIRP >> 0.01 1469.51 us 231.67 us 440065.71 us 22845 >> MKNOD >> 0.01 44229.36 us 75.58 us 1803963.42 us 871 >> FTRUNCATE >> 0.01 84433.20 us 149.47 us 3614869.24 us 1001 >> SETATTR >> 0.02 418.11 us 13.39 us 3515000.81 us 243820 >> FLUSH >> 0.03 1108.47 us 14.77 us 1658743.41 us 130647 >> GETXATTR >> 0.03 562.86 us 26.74 us 5043949.33 us 297799 >> STATFS >> 0.03 845.50 us 48.16 us 1998938.03 us 223680 >> OPEN >> 0.05 2236.13 us 56.08 us 5295998.22 us 120682 >> XATTROP >> 0.16 890.82 us 43.51 us 3653292.88 us 1012585 >> FSTAT >> 0.34 2974.20 us 12.92 us 7782497.74 us 642555 >> ENTRYLK >> 0.47 135831.21 us 70.35 us 11033867.90 us 19800 >> UNLINK >> 0.57 954.05 us 21.50 us 4553393.38 us 3414605 >> LOOKUP >> 1.79 14461.54 us 13.45 us 32841452.98 us 702915 >> INODELK >> 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 >> READ >> 5.37 443.04 us 11.53 us 32863652.53 us 68909131 >> FINODELK >> 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 >> FXATTROP >> 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 >> WRITE >> 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 95492 >> UPCALL >> >> Duration: 484169 seconds >> Data Read: 167149718723 bytes >> Data Written: 1177141649872 bytes >> >> Interval 43 Stats: >> Block Size: 1b+ 256b+ >> 512b+ >> No. of Reads: 0 6 >> 4 >> No. of Writes: 12 4 >> 252 >> >> Block Size: 1024b+ 2048b+ >> 4096b+ >> No. of Reads: 0 0 >> 5668 >> No. of Writes: 100 34 >> 147357 >> >> Block Size: 8192b+ 16384b+ >> 32768b+ >> No. of Reads: 1178 783 >> 1215 >> No. of Writes: 86014 17318 >> 8687 >> >> Block Size: 65536b+ 131072b+ >> No. of Reads: 264 4109 >> No. of Writes: 8617 36317 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 16 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 665 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 52 >> RELEASEDIR >> 0.00 866.18 us 849.57 us 882.78 us 2 >> TRUNCATE >> 0.00 1016.14 us 868.35 us 1182.48 us 4 >> RENAME >> 0.00 455.65 us 41.13 us 1679.37 us 20 >> READDIR >> 0.00 373.33 us 173.54 us 538.40 us 28 >> UNLINK >> 0.00 722.01 us 635.91 us 853.71 us 16 >> CREATE >> 0.00 335.52 us 38.15 us 1381.97 us 54 >> READDIRP >> 0.00 39218.92 us 214.34 us 78223.50 us 2 >> STAT >> 0.00 1730.77 us 2.59 us 84964.69 us 52 >> OPENDIR >> 0.00 1304.96 us 393.64 us 29724.09 us 80 >> MKNOD >> 0.01 422.75 us 22.85 us 134336.60 us 635 >> FLUSH >> 0.01 699.21 us 37.54 us 137813.64 us 732 >> STATFS >> 0.01 468.63 us 23.71 us 260961.83 us 1141 >> ENTRYLK >> 0.04 2041.83 us 69.80 us 163743.98 us 649 >> OPEN >> 0.08 1253.65 us 72.52 us 290508.15 us 2354 >> FSTAT >> 0.09 6913.19 us 69.01 us 212479.24 us 472 >> XATTROP >> 0.13 3558.26 us 32.70 us 195896.10 us 1317 >> INODELK >> 0.13 9793.37 us 36.11 us 212755.58 us 499 >> GETXATTR >> 0.34 1615.15 us 55.42 us 711310.28 us 7797 >> LOOKUP >> 4.92 13025.92 us 59.83 us 1061585.06 us 13884 >> READ >> 5.26 955.95 us 16.81 us 1069148.49 us 202174 >> FINODELK >> 23.28 2809.44 us 137.21 us 1134076.13 us 304713 >> WRITE >> 26.46 4124.05 us 43.30 us 1142167.27 us 235954 >> FXATTROP >> 39.23 5195.62 us 65.47 us 1250469.35 us 277632 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 365 >> UPCALL >> >> Duration: 1132 seconds >> Data Read: 668951066 bytes >> Data Written: 7495356076 bytes >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E… >> >

5 7

[Call to Action] oVirt 4.5 localization updated in zanata
by Sandro Bonazzola 24 Feb '22

24 Feb '22

Hi, https://zanata.ovirt.org/ has been updated with new text to be translated. If you want to help with this, please follow https://ovirt.org/develop/localization.html The ovirt-web-ui and ovirt-engine-ui-extensions projects lack Italian, Czech and Turkish translation. The oVirt engine project needs some touch on all the translations but especially on Italian, Czech, Russian and Turkish Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA sbonazzo(a)redhat.com Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.

1 0

Migrate Hosted-Engine from one NFS mount to another NFS mount
by Brian Ismay 24 Feb '22

24 Feb '22

Hello, I am currently running Ovirt 4.2.5.3-1.el7 with an Hosted-Engine running on nfs-based storage. I am needing to migrate the backend NFS store for the Hosted-Engine to a new share. Is there specific documentation for this procedure? I see multiple references to a general procedure involving backing-up the existing install and then re-installing and restoring the backup. Is there any way to make a more clean change of the configuration? Thanks, Brian Ismay

2 1

Re: VMs losing network interfaces
by Strahil Nikolov 23 Feb '22

23 Feb '22

That's a good start. Are you building your VMs from template ?Any chance there is another system (outside oVirt ?) with the same MAC ? you can run arping and check if you get response from more than 1 system. Best Regards,Strahil Nikolov On Mon, Feb 21, 2022 at 11:15, jb<jonbae77(a)gmail.com> wrote: Thank you Nikolov, I have descript the problem a bit wrong. In the UI is see the interface, with virsh dumpxml I get: <interface type='bridge'> <mac address='00:1a:4a:16:01:83'/> <source bridge='ovirtmgmt'/> <target dev='vnet9'/> <model type='virtio'/> <driver name='vhost' queues='4'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <mtu size='1500'/> <alias name='ua-c8a50041-2d13-456d-acb6-b57fdaea434b'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> lspci on the VM show also the interface and I can also bring the interface up again with: ifup enp1s0. So it only loose the connection, not the interface. When I restart the VM I get this log message: /var/log/syslog:Feb 21 10:11:29 tv-planer sh[391]: ifup: failed to bring up enp1s0 /var/log/syslog:Feb 21 10:11:29 tv-planer systemd[1]: ifup(a)enp1s0.service: Main process exited, code=exited, status=1/FAILURE /var/log/syslog:Feb 21 10:11:29 tv-planer systemd[1]: ifup(a)enp1s0.service: Failed with result 'exit-code'. But the VM got his IP and is reachable. Am 20.02.22 um 14:06 schrieb Strahil Nikolov: Do you see all nic in the UI ? What type are they ? Set this alias on the Hypervisors: alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' and then use 'virsh dumpxml name-of-vm' to identify how many nics the vm has . If gou got the correct settings in ovirt, use 'lspci -vvvvv' . Best Regards, Strahil Nikolov On Sun, Feb 20, 2022 at 11:32, Jonathan Baecker <jonbae77(a)gmail.com> wrote: Hello everybody, I have here a strange behavior: We have a 3 node self hosted cluster with around 20 VMs running on it. Since longer I had the problem with one VM that after some days it lose the network interface. But because this VM was only for testing I was to lazy to dive more in, to figure out what is happen. Now I have a second VM, with the same problem and this VM is more important. Both VMs running debian 10 and use cifs mounts, so maybe that is related? Have some one of you seeing this behavior? And can give me a hint, how I can fix this? At the moment I can't provide a log file, because I didn't know the exact time, when this was happen. And I also don't know, if the problem comes from ovirt or from the operating system inside the VMs. Have a nice day! Jonathan _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DOXZRQ55LFPNK…

2 4

hosted engine deployment (v4.4.10) - TASK Check engine VM health - fatal FAILED
by Charles Stellen 23 Feb '22

23 Feb '22

Dear Ovirt Hackers, sorry: incidently send to devel(a)ovitr.org we are dealing with hosted engine deployment issue on a fresh AMD EPYC servers: and we are ready to donate hardware to Ovirt community after we pass this issue ( :-) ) 0/ base infra: - 3 identical physical servers (produced in 2021-4Q) - fresh, clean and recent version of centos 8 stream installed (@^minimal-environment) - servers are interconnected with cisco switch, each other are network visible, all with nice internet access (NAT) 1/ storage: - all 3 servers/nodes host nice and clean glusterfs (v9.5) and volume "vol-images01" is ready for VM images - ovirt hosted engine deployment procedure: - easily accept mentioned glusterfs storage domain - mount it during "hosted-engine --deploy" with no issue - all permissions are set correctly at all glustrfs nodes ("chown vdsm.kvm vol-images01") - no issue with storage domain at all 2/ ovirt - hosted engine deployment: - all 3 servers successfully deployed recent ovirt version with standart procedure (on top of minimal install of centos 8 stream): dnf -y install ovirt-host virt-host-validate: PASS ALL - at first server we continue with: dnf -y install ovirt-engine-appliance hosted-engine --deploy (pure commandline - so no cockpit is used) DEPLOYMENT ISSUE: - during "hosted-engine --deploy" procedure - hosted engine becomes temporairly accessible at:https://server01:6900/ovirt-engine/ - with request to manualy set "ovirtmgmt" virtual nic - Hosts > server01 > Network Interfaces > [SETUP HOST NETWORKS] "ovirtmgmt" dropped to eno1 - [OK] - than All pass fine - and host "server01" becomes Active - back to commandline to Continue with deployment "Pause execution until /tmp/ansible.jksf4_n2_he_setup_lock is removed" by removing the lock file - deployment than pass all steps_until_ "[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check engine VM health]" ISSUE DETAILS: new VM becomes not accessible in the final stage - as it should be reachable at its final IP: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fail if Engine IP is different from engine's he_fqdn resolved IP] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM IP address is while the engine's he_fqdn ovirt-engine.mgmt.pss.local resolves to 10.210.1.101. If you are using DHCP, check your DHCP reservation configuration"} - problem is, that even if we go with "Static" IP (provided during answering procedure) or with "DHCP" way (with properly set DHCP and DNS server responding with correct IP for both WE STUCK THERE WE TRYIED: - no success to connect to terminal/vnc of running VM "HostedEngine" to figure out the internal network issue any suggestion howto "connect" into newly deployed UP and RUNNING HostedEngine VM? to figure out eventually manualy fix the internal network issue? Thank You all for your help Charles Stellen PS: we are advanced in Ovirt deployment (from version 4.0), also we are advanced in GNU/Linux KVM based virtualisation for 10+ years, so any suggests or any details requested - WE ARE READY to provide online debuging or direct access to servers is not a problem PPS: after we pass this deployment - and after decomissioning procedure - we are ready to provide older HW to Ovirt community

6 14

how to convert centos 8 to centos 8 Stream
by Adam Xu 23 Feb '22

23 Feb '22

Dear ovirt community, I can't upgrade ovirt to version 4.4.10 because of EOL of centos8. How can we convert centos 8 to centos 8 stream? Thanks.

2 3

help using nvme/tcp storage with cinderlib and Managed Block Storage
by muli＠lightbitslabs.com 23 Feb '22

23 Feb '22

Hi everyone, We are trying to set up ovirt (4.3.10 at the moment, customer preference) to use Lightbits (https://www.lightbitslabs.com) storage via our openstack cinder driver with cinderlib. The cinderlib and cinder driver bits are working fine but when ovirt tries to attach the device to a VM we get the following error: libvirt: error : cannot create file '/var/run/libvirt/qemu/18-mulivm1.dev/mapper/': Is a directory We get the same error regardless of whether I try to run the VM or try to attach the device while it is running. The error appears to come from vdsm which passes /dev/mapper as the prefered device? 2022-02-22 09:50:11,848-0500 INFO (vm/3ae7dcf4) [vdsm.api] FINISH appropriateDevice return={'path': '/dev/mapper/', 'truesize': '53687091200', 'apparentsize': '53687091200'} from=internal, task_id=77f40c4e-733d-4d82-b418-aaeb6b912d39 (api:54) 2022-02-22 09:50:11,849-0500 INFO (vm/3ae7dcf4) [vds] prepared volume path: /dev/mapper/ (clientIF:510) Suggestions for how to debug this further? Is this a known issue? Did anyone get nvme/tcp storage working with ovirt and/or vdsm? Thanks, Muli

4 7

Migrating VMs from 4.3 Ovirt Env to new 4.4 Env
by Abe E 23 Feb '22

23 Feb '22

So as title states I am moving VMs from an old system of ours with alot of issues to a new 4.4 HC Gluster envr although it seems I am running into what I have learnt is a 4.3 bug of some sort with exporting OVAs. Some of my OVAs are showing large GB sizes although their actual size may be 20K so they are showing no boot device as well, theres really no data there. Some OVAs are fine but some are broken. So I was wondering if anyone has dealt with this and whats the best way around this issue? I have no access remotely at the moment but when I get to them, I read somewhere its better to just detach disks and download them to a HDD for example and build new VMs and upload those disks and attach them instead this way? Please do share if you have any better ways, usually I just Export OVA to an external HDD, remount to new ovirt and import OVA but seems a few of these VMs out of alot did not succeed and I am running out of downtime window. Thanks a bunch, I have noticed repeated names helping me, I am very grateful for your help.

3 5

Re: oVirt alternatives
by Thomas Hoberg 22 Feb '22

22 Feb '22

That's exactly the direction I originally understood oVirt would go, with the ability to run VMs and container side-by-side on the bare metal or nested with containers inside VMs for stronger resource or security isolation and network virtualization. To me it sounded especially attractive with an HCI underpinning so you could deploy it also in the field with small 3 node clusters. But combining all those features evidently comes at too high a cost for all the integration and the customer base is either too small or too poor: the cloud players are all out on making sure you no longer run any hardware and then it's really just about pushing your applications there as cloud native or "IaaS" compatible as needed. E.g. I don't see PCI pass-through coming to kubevirt to enable GPU use, because it ties the machine to a specific host and goes against the grain of K8 as I understand it. Memory overcommit is quite funny, really, because it's the same issue as the original virtual memory: essentially you lie to your consumer about the resources available and then swap pages forth and back in an attempt to make all your consumers happy. It was processes for virtual memory, it's VMs now for the hypervisor and in both cases it's about the consumer and the provider not continously negotiating for the resources they need and the price they are willing to pay. That negotiation is always better at the highest level of abstraction, the application itself, which why implementing it at the lower levels (e.g. VMs) becomes less useful and needed. And then there is technology like CXL which essentially turns RAM in to a fabric and your local CPU will just get RAM from another piece of hardware when your application needs more RAM and is willing to pay the premium something will charge for it. With that type of hardware much of what hypervisors used to do goes into DPUs/IPUs and CPUs are just running applications making hypercalls. The kernel is just there to bootstrap. Not sure we'll see that type of hardware at home or in the edge, though...

4 7

Gluster Performance issues
by Alex Morrison 22 Feb '22

22 Feb '22

Hello All, We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements "gluster volume info" and "gluster volume profile vmstore info" is below ========================================================================================================================= -Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, ========================================================================================================================= -Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, ========================================================================================================================= [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off ========================================================================================================================= [root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687 Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687 Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes

1 0

Unable to update self hosted Engine due to missing mirrors
by jonas＠fleckmail.de 22 Feb '22

22 Feb '22

Hi together, I already checked the other Threads here :) I know that there was an issue with old CentOS 8 repos. Currently Im trying so update our self hosted engine as always but I get the following error: Failed to download metadata for repo 'ovirt-4.4-centos-gluster8': Cannot prepare internal mirrorlist: No URLs in mirrorlist Is there any "official" advise what I have to change in my repo lists? I don't want to break something. Thank you very much :)

8 18

live storage migration & export speed
by Роман Черкалин 22 Feb '22

22 Feb '22

I'm even wondering who has what speed of export VM or live storage migration? I just don't understand how to increase it. ZFS replication goes 300-400MB/s with the same load. With regards, Roman Cherkalin

2 1

Broke my GlusterFS somehow
by Abe E 22 Feb '22

22 Feb '22

So upon changing my ovirt nodes (3Hyperconverged Gluster) as well as my engines hostname without a hitch I had an issue with 1 node and somehow I did something that broke its gluster and it wouldnt activate, So the gluster service wont start and after trying to open the node from webgui to see what its showing in its virtualization tab I was able to see that it allows me to run the hyperconverged wizard using the existing config. Due to this i lost the engine because well the 3rd node is just arbiter and node 2 complained about not having shared storage. This node is the one which I built ovirt gluster from so i assumed it would rebuild its gluster.. i accidentally clicked cleanup which got rid of my gluster brick mounts :)) then I tried to halt it and rebuild using existing configuration. Here is my issue though, am I able to rebuild my node? This is a new lab system so I believe i have all my vms still on my external HDDs. If I can restore this 1 node and have it rejoin the gluster then great, otherwise whats the best route using the webgui (I am remote at the moment) to just wipe all 3 nodes and start all over again and work it slowly? Is it simply deleting the partitions for the ovirt glusters on each node enough to let me rebuild ?

4 7

Install hosted engine using fcoe
by kyles＠aasteel.com 22 Feb '22

22 Feb '22

We're building a new cluster, but it utilizes fcoe for our storage. Is there a way to install the hosted engine using fcoe? I am unable to see our LUNs in the ovirt node currently as fcoe isn't enabled.

2 1

ansible ovirt_template: follow as replacement of nested_attributes
by Nathanaël Blanchet 21 Feb '22

21 Feb '22

Hello, I was familiar with the ansible fetch_nested/nested_attributes to get additionnal information from API. Recently it has been deprecated in order to be replaced by the "follow(s)" verb. I understand it will follow the links into the schema definition of the object like: <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/permissions" rel="permissions"/> <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/tags" rel="tags"/> <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/cdroms" rel="cdroms"/> <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/nics" rel="nics"/> <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/watchdogs" rel="watchdogs"/> <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/diskattachments" rel="diskattachments"/> <link href="/ovirt-engine/api/templates/e020281a-f5f0-4d98-91e3-4f3a9fc069b1/graphicsconsoles" rel="graphicsconsoles"/> but I can't get fetch nested (name of cluster) of : <cluster href="/ovirt-engine/api/clusters/bbeb38a4-79f0-11ec-86cb-00163e6d6a50" id="bbeb38a4-79f0-11ec-86cb-00163e6d6a50"/> I was used to do this query with: fetch_nested: yes nested_attribute: - name Is it possible to do the same with "follow(s)"? -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

1 1

VMs losing network interfaces
by Jonathan Baecker 21 Feb '22

21 Feb '22

Hello everybody, I have here a strange behavior: We have a 3 node self hosted cluster with around 20 VMs running on it. Since longer I had the problem with one VM that after some days it lose the network interface. But because this VM was only for testing I was to lazy to dive more in, to figure out what is happen. Now I have a second VM, with the same problem and this VM is more important. Both VMs running debian 10 and use cifs mounts, so maybe that is related? Have some one of you seeing this behavior? And can give me a hint, how I can fix this? At the moment I can't provide a log file, because I didn't know the exact time, when this was happen. And I also don't know, if the problem comes from ovirt or from the operating system inside the VMs. Have a nice day! Jonathan

3 2

Re: oVirt alternatives
by Nathanaël Blanchet 21 Feb '22

21 Feb '22

2 1

oVirt alternatives
by Thomas Hoberg 21 Feb '22

21 Feb '22

There is unfortunately no formal announcement on the fate of oVirt, but with RHGS and RHV having a known end-of-life, oVirt may well shut down in Q2. So it's time to hunt for an alternative for those of us to came to oVirt because they had already rejected vSAN or Nutanix. Let's post what we find here in this thread.

13 21

unable to download and install ldap package to integrate with AD. oVirt Engine version - ovirt-engine-4.4.9.5-1
by umakanta.samantaray＠broadcom.com 21 Feb '22

21 Feb '22

When trying to installing the package ovirt-engine-extension-aaa-ldap-setup, receiving below error message, can you please help # sudo yum install ovirt-engine-extension-aaa-ldap-setup Last metadata expiration check: 0:08:27 ago on Monday 21 February 2022 11:41:51 AM IST. Error: Problem: cannot install the best candidate for the job - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.5-1.el8.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) # sudo yum install ovirt-engine-extension-aaa-ldap-setup --nobest Last metadata expiration check: 0:16:27 ago on Monday 21 February 2022 11:41:51 AM IST. Error: Problem: conflicting requests - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.0-1.el8.noarch - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.1-1.el8.noarch - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.2-1.el8.noarch - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.3-1.el8.noarch - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.4-1.el8.noarch - nothing provides python3-ldap needed by ovirt-engine-extension-aaa-ldap-setup-1.4.5-1.el8.noarch (try to add '--skip-broken' to skip uninstallable packages) # sudo yum install ovirt-engine-extension-aaa-ldap-setup --skip-broken Last metadata expiration check: 0:16:58 ago on Monday 21 February 2022 11:41:51 AM IST. Dependencies resolved. Nothing to do. Complete!

1 0

Re: ovirtmgmt VLAN and IP Change
by Abe E 20 Feb '22

20 Feb '22

Your method worked ! I additionally had to run the engine rename service script to change the hostname of the engine as well. My only issue now is after readding all the nodes with new hostnames my main host node 1 will not allow me to remove and re-add as a new hostname as there is a gluster volume on it. It will not let me remove to readd because theres a gluster on the volume, have you run into this?

2 1

Re: ovirtmgmt VLAN and IP Change
by Abe E 19 Feb '22

19 Feb '22

How did you do this if the host had a gluster volume on it though? Mine does not let me remove due to that.

2 1

ovirtmgmt VLAN and IP Change
by Abe E 19 Feb '22

19 Feb '22

Hello All, So due to some issues with provisioning new networks in our lab, I have to use a different VLAN in for my ovirt cluster. Cluster -> 3 Hosts (Hyperconverged) What I want to do is essentially change the Ovirtmgmt interface to another vlan but also change the IP Schemes, I wont be needing to change the storage network vlans. I am just wondering how this is done safely. Can I through the engine, put the host 1 by 1 into maint mode and manually adjust the IPs and apply and reconfigure the switch port? What about the IP that the engine has set or would I change that manually through console once I change the ovirtmgmt vlan on all the nodes?

3 3

Random reboots
by Pablo Olivera 17 Feb '22

17 Feb '22

Hi community, We're dealing with an issue as we occasionally have random reboots on any of our hosts. We're using ovirt 4.4.3 in production with about 60 VM distributed over 5 hosts. We've a virtualized engine and a DRBD storage mounted by NFS. The infrastructure is interconnected by a Cisco 9000 switch. The last random reboot was yesterday February 14th at 03:03 PM (in the log it appears as: 15:03 due to our time configuration) of the host: 'nodo1'. At the moment of the reboot we detected in the log of the switch a link-down in the port where the host is connected. I attach log of the engine and host 'nodo1' in case you can help us to find the cause of these random reboots. Thanks in advance. Pablo.

5 11

Re: Enable Power Management Ovirt 4.3
by emiliano.pozzessere＠satservizi.eu 17 Feb '22

17 Feb '22

Thanks. Best regards

1 0

Re: Enable Power Management Ovirt 4.3
by emiliano.pozzessere＠satservizi.eu 17 Feb '22

17 Feb '22

Hi Angus, is kdump integration recommended to enable it?

2 1

Re: Enable Power Management Ovirt 4.3
by emiliano.pozzessere＠satservizi.eu 17 Feb '22

17 Feb '22

Ok thanks for support :-) Best regards

1 0

Re: Enable Power Management Ovirt 4.3
by emiliano.pozzessere＠satservizi.eu 17 Feb '22

17 Feb '22

Thanks for reply Angus, IPMI/DCMI over LAN is enable On ILO4 I created account for Ovirt with full credential and now it's works with ILO4 The ports and protocol are enable, but Ovirt works only ip_ssh and ILO4, no ipmilan. Do you know which keys I have to set in the option section of the agent fence on Ovirt? Thanks

2 1

Cannot add Virtual Disk. Disk configuration (RAW Sparse backup-None) is incompatible with the storage domain type.
by nicolas＠devels.es 17 Feb '22

17 Feb '22

Hi, We're using oVirt 4.4.8.6. We make an intensive use of the VM portal because we've hundreds of students creating their own VMs. Recently, one of the professors reported that they are encountering an error when adding a disk to a newly created VM. They are creating ISO based VMs (CentOS-8-Stream in this case), everything goes smoothly but when adding a thin-provisioned disk, this error shows up: 2022-02-17 09:56:28,073Z INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-39078) [0332e7b6-80b1-48e9-b849-80698f2ce7ab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e4a02ab9-31e4-4e8c-8999-91700263ff08=VM_DISK_BOOT]', sharedLocks='[e4a02ab9-31e4-4e8c-8999-91700263ff08=VM]'}' 2022-02-17 09:56:28,446Z WARN [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-39078) [0332e7b6-80b1-48e9-b849-80698f2ce7ab] Validation of action 'AddDisk' failed for user aluX@domain-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_CONFIGURATION_NOT_SUPPORTED,$volumeFormat RAW,$volumeType Sparse,$backup None 2022-02-17 09:56:28,446Z INFO [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-39078) [0332e7b6-80b1-48e9-b849-80698f2ce7ab] Lock freed to object 'EngineLock:{exclusiveLocks='[e4a02ab9-31e4-4e8c-8999-91700263ff08=VM_DISK_BOOT]', sharedLocks='[e4a02ab9-31e4-4e8c-8999-91700263ff08=VM]'}' 2022-02-17 09:56:28,504Z ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-39078) [] Operation Failed: [Cannot add Virtual Disk. Disk configuration (RAW Sparse backup-None) is incompatible with the storage domain type.] However, changing the provisioning to thick does work and the disk can be added. I found [1] which talks about this but I'm not sure if it's the same issue, nor it has a solution yet. Is this a known bug? Does it have any workaround beyond creating thick-provisioned disks? Thanks. Nicolás [1]: https://access.redhat.com/solutions/6022811

2 1

Enable Power Management Ovirt 4.3
by emiliano.pozzessere＠satservizi.eu 17 Feb '22

17 Feb '22

Hi guys, I would like enable Power Management on Ovirt 4.3. My Hosts are HP with ILO4, I configured on ILO4 account for Ovirt. On the host enable Power Management, flag on Kdump Integration and I added the fence agent, I put the user and password and set up ILO4 but the test fails, if I put ilo_ssh it works. But I don't know which keys I have to enter and with which values (if necessary) Can anyone explain to me? Thanks

2 1

Enable Power Management Ovirt 4.3
by emiliano.pozzessere＠satservizi.eu 17 Feb '22

17 Feb '22

Good morning, I need a clarification on enabling Power Management. I have an Ovirt 4.3 and some hosts with ILO4, if I enable power management on ILO4 the test fails if instead I enable it on ilo_ssh it works, but after I have enabled it I don't know which keys I have to set, can you help me? Thanks

1 0

Centos 8.3 ovirt4.4.5 to Centos Stream 8.6 ovirt 4.4.10 migiration success
by Ismet Sonmez 16 Feb '22

16 Feb '22

Hello everyone We did a nice update over the weekend. Everything is working fine. Thank you to the whole team for developing such a project. Best regards

1 1

Xcp-ng, first impressions as an oVirt HCI alternative
by Thomas Hoberg 16 Feb '22

16 Feb '22

Comments & motivational stuff were moved to the end... Source/license: Xen the hypervisor is moved to the Linux foundation. Perpetual open source, free to use. Xcp-ng is a distribution of Xen, produced by a small French company based on Xen using (currently) a Linux 4.19 LTS kernel and an EL7 frozen userland from July 2020: they promise open source and free to use forever Mode of operation: You install Xcp-ng on your (bare metal) hardware. You manage nodes via XenOrchestrator, which is a big Node.js application you can run in pretty much any way you want. Business model: You can buy support for Xcp-ng at different levels of quality. XenOrchestrator as AN APPLIANCE exposes different levels of functionality depending on the level of support you buy. But you get the full source code of the appliance and can compile it yourself to support the full set of qualities. There is a script out there, which allows you to auto-generate the appliance with a single command. In short you are never forced to pay, but better help can be purchased. How does it feel to a CentOS/RHEL user? The userland on the nodes is EL7, but you shouldn't touch that. CLI is classic Xen, nothing like KVM or oVirt. I guess libVirt and virsh should be similar, if they live up to their promise at all. Standard user-land on Orchestrator appliance is Debian, but you can build it on pretty much any Linux with that script: all interaction is meant to be done via Web-UI. Installation/setup: There is an image/ISO much like the oVirt node image. Based on a Linux 4.19 LTS kernel and an EL7 frozen userland from July 2020 and a freshly maintained Xen with tools. Installation on bare metal or VMs (e.g. for nested experimentation) is a snap, HCL isn't extraordinary. I'm still fighting to get the 2.5/5GBit USB3 NIC working that I like using for my smallest test systems. A single command on one node will download the "free"-Orchestrator appliance (aka Xoa) and install it as a VM on that node. It's installed as auto-launch and just point your brower to its IP to start with the GUI. There is various other ways to build or run the GUI, which can be run on anything remotely Linux, within the nodes or outside: more on this in the next section. The management appliance (Xoa) will run with only 2GB of RAM and 10GB of disk for a couple of hosts. You grow to dozens of hosts, give a little more RAM and it will be fine. Compare to the oVirt management engine it's very, very light seems to have vastly less parts that can break. And if it does, that doesn't matter, because it is pretty much stateless. E.g. pool membership and configuratoin is on the nodes, so if you connect from another Xoa they will just carry over. Ditto storage, that configuration which oVirt keeps in the management engines Postgres database, is on the nodes in Xcp and can be changed by any connected Xoa. Operation: Xen nodes are much more autonomous then oVirt hosts. The use whatever storage they might have locally, or attached via SAN/NAS/Gluster[!!!] and others.They will operate without a management engine much like "single node HCI oVirt" or they can be joined into a pool, which opens up live migration and HA. A pool is created by telling a node that it's the master now and then adding other nodes to join in. The master can be changed and nodes can be moved to other pools. Adding and removing nodes to a pool is very quick and easy and it's the same for additional storage repositories: Any shared storage added to any node is immediately visible to the pool and disks can be flipped between local and shared storage very easily (I haven't tried live disk moves, but they could work). Having nodes in a pool qualfies them for live migration (CPU architecture caveats apply). If storage is local, it will move with the VM, if storage is shared, only RAM will move. You can also move VMs not sharing a pool and even across different x86 variants (e.g. AMD and Intel), when VMs are down. If you've ever daddled with "Export domains" or "Backup domains" in oVirt, you just can't believe how quick and easy these things are in Xcp-ng. VMs and their disks can be moved, copied, cloned, backed-up and restore with a minimum of fuzz including continuous backups on running machines. You can label machines as "HA" so they'll always be restarted elsewhere, should a host go down. You can define policies for how to balance workloads across hosts and ensure that HA pairs won't share a host, pretty similar to oVirt. The "free" Xoa has plenty of "upgrade!" buttons all over the place. So I went ahead and build an appliance from source, that doesn't have these restrictions, just to see what that would get me. With this script here: https://github.com/ronivay/XenOrchestraInstallerUpdater you can build the Xoa on any machine/VM you happen to be running with one of the many supported Linux variants. I build one variant to run as a VM on Xcp-ng and I used another to run on an external machine: All three Xoas don't step on each others toes more than you make them: very, very cool! The built-in hypervisor interfaces and VIO drivers of almost any modern 64-Bit Linux make VMs very easy. For Windows guests there are drivers available, which make things as easy as with KVM. Unfortunately they don't seem to be exactly the same, so you may do better to remove guest drivers on machines you want to move. BTW good luck on trying to import OVAs exported from oVirt: Those contain tags, no other hypervisor wants to accept, sometimes not even oVirt itself. Articles on the forum suggest running CloneZilla at both ends for a strorage-less migration of VMs. I have not yet tested nested virtualization inside Xcp-ng (Xcp-ng works quite well in a nested environment), nor have I done tests with pass-through devices and GPUs. All of that is officially supported, with the typical caveats in case of older Nvidia drivers. So far every operation was quick and every button did what I expected it to do. When things failed, I didn't have to go through endless log files to find out what was wrong: the error messages were helpful enough to find the issues so far. Caveats: I haven't done extensive failure testing yet, except one: I've been running four host nodes as VMs on VMware workstation, with XCP-ng VMs being nested inside. At one point I managed to run out of storage and I had to shut down all "bare metal VMs" hard. Everything came back clean, only the migration project that had tripped the storage failure had to be restarted, after I created some space. There is no VDO support in the kernel provided. It should be easy enough to add given the sources and build scripts. But the benefit and the future of VDO are increasingly debated. The VMs use the VDI disk format, which is less powerful/flexible as CQOW with regards to thin allocation and trimming. It's on the roadmap, not yet in the product. Hyperconverged Storage: Xcp-ng so far doesn't come with a hyperconvered storage solution built in. Like so many clusters it moves the responsability to your storage layer (could be Gluster...). They are in the process of making LINSTOR a directly supported HCI option, even with such features as dispersed (erasure-coded) volumes to manage the write amplification/resilience as the node numbers increase beyond three. That's not ready today and they seem to take their sweet time about it. They label it "XOASAN" and want to make that a paid option. But again, the full source code is there and in the self-compiled Xoa appliance you should be able to use it for free. The current beta release only supports replicated mode and only up to four nodes. But it seems to work reliably. Write amplification is 4x, so bandwidth drops to 25% and is limited to the network speed, but reads will go to the local node at storage hardware bandwidths. The 2, 3 and 4 node replicated setup works today with the scripts they provide. That's not quite as efficient as the 2R+1A setup in oVirt, but seems rock solid and just works, which was never that easy in oVirt. Hyperconverged is very attractive at 3 nodes, because good fault resilience can't be done any cheaper. But the industry agrees that it tends to lose financial attraction when you grow to dozens of machines and nobody in is right mind would operate a real cloud using HCI. But going from 3 to say two dozen should be doable and easy: it never was for oVirt. XCP-ng or rather LINSTOR won't support any number or storage nodes or easy linear increases like Gluster could (in theory). So far it's only much, much better at getting things going. Forum and community: The documentation is rather good, but can be lighter on things which are "native Xen", as that would repeat a lot of the effort already expended by Citrix. It's good that I've been around the block a couple of times already since VM/370, but there are holes or missing details when you need to ask questions. The community isn't giant, but comfortably big enough. The forum's user interface is vastly better than this one, but then I've never seen something as slow as ovirt.org for a long time. Technical questions are answered extremely quickly and mostly by the staff from the small French company themselves. But mostly it's much easier to find answers to questions already asked, which is the most typical case. The general impression is that there are much fewer moving parts and things things that can go wrong. There is no Ansible, not a single daemon on the nodes, a management engine that seems very light and with minimal state that doesn't need a DBA to hold and manage it. The Xen hypervisor seems much smarter than KVM on its own and the Xoa has both a rich API to do things, but also offers an API to next-level management. It may have less overall features than oVirt, but I haven't found anything that I really missed. It's much easier and quicker to install and operate with nothing but GUI which is a godsent: I want to use the farm, not spend my time managing it. Motivational rant originally at the top... tl;dr My original priorities for chosing oVirt were: 1. CentOS as RHEL downstream -> stable platform, full vendor vulnerability management included, big benefit in compliance 2. Integrated HCI -> Just slap a couple (3/6/9) of leftover servers together for something fault resilient, no other components required, quick start out of the box solution with a few clicks in a GUI 3. Fully open source (& logs), can always read the source to understand what's going on, better than your typical support engineer 4. No support or license contract, unless you want/need it, but the ability to switch that on when it paid for itself The more famous competitors, vSphere and Nutanix didn't offer any of that. (Citrix) Xen I excluded, because a) Xen seemd "old school+niche" compared to KVM b) Citrix reduced "free" to "useless" I fell in love with Gluster from its design: it felt like a really smart solution. I fell out with Gluster from its operation and performance: I can't count how many times I had to restart daemons and issue "gluster heal" commands to resettle things after little more than a node update. I rediscovered Xcp-ng when I discovered that HCI and RHV had ben EOLd.

7 9

Restore from backup, consistent view between ovirt DB and Libvirt/QEMU
by Volenbovskyi, Konstantin 16 Feb '22

16 Feb '22

Hi, Let’s say that backup of ovirt engine is done once every day, for example 6 AM using engine-backup tool. Then during the day the changes are made that not only influence the database, but actually influence Libvirt/KVM/QEMU (create/delete VMs,snapshots, disks, ). Now let’s say at 18:00 an engineer restores from the backup. How the ovirt will be ‘synchronized’ to Libvirt (/QEMU)? Of course, it is OK to expect that system is being rolled back, but in that scenario there is nothing that I can imagine that will synchronize Ovirt database state to Libvirt/KVM/QEMU state. And once anything on (migrate/stop/start) – it will just fail in many cases because of inconsistency. In 4.2 documentation https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… there is sentence like: “If migration is enabled at cluster level, virtual machines are automatically migrated to another host in the cluster; as a result, it is recommended that host reinstalls are performed at a time when the host’s usage is relatively low.“ Let’s say that there are 5 hosts (numbered from 1 to 5), 3 of them are hosted-engine (let’s say 1 to 3). Then first of all, person should find a brand new host (host 6), then install HE on that. OK, I understand that HE can be then reinstalled on hosts 1 to 3. But it when it comes VMs- if really migration of VMs can work, considering inconsistency above? Eg. snapshot that is already in QEMU (done at 11:00) and the volume chain is updated is gone from database (because it is restored to 6 AM) And in general, anything should be done towards non-HE hosts? So far the restore seems to be very intrusive and quite error-prone in two parts: first part is actual handling of VMs. And second part when ovirt-engine gets configured practically from scratch (through that dialog) BR, Konstantin

1 0

Re: vm seal
by Nathanaël Blanchet 16 Feb '22

16 Feb '22

2 4

Re: Problems configuring iscsi domain on ovirt 4.4.4.7-1.el8 / centos 8
by David Johnson 16 Feb '22

16 Feb '22

Ok. The nfs share with the vdsm group is working. But the intent is to switch to iSCSI. The little bit of documentation about iSCSI mentions nothing about vdsm or group 36. That gives me something to work with at least. Any ideas about how to clean up the mess left by the first attempt? I'm not finding the records for botched domain creation in the engine database, but the GUI shows them and complains that the iSCSI device is registered twice. On Wed, Feb 16, 2022, 2:06 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: > To be honest, I just asked from curiosity. > In NFS it's mandatory to use group 36, but this is the linux group. > > I guess in TrueNAS it's something else, as I never configured group in > targetcli. > > The flow is usually: > - Target server listens on multiple IPs > - Client makes discovery on a single ip > - client logs in (lsscsi is the simplest approach to verify it) > - Client discovers and logs on other IPs that the target server uses > - Multipath aggregates the links > > As Multipath has a timeout of 20s while iscsi client uses 120s by default, > I usually change the client to use 15s timeout. > > When a path (for eample 1 switch is dead) is down, multipath will try > another link to reach the target server. > > It's nice there is a vdsm tool to configure those . > > WARNING: Never set clients with the same IQN (defined in a file in > /etc/iscsi...somethig -> see with rpm -ql iscsi-utils) > > Best Regards, > Strahil Nikolov > > On Wed, Feb 16, 2022 at 9:18, David Johnson > <djohnson(a)maxistechnology.com> wrote: > I used group 1 because nothing said what group to use, and googling "ovirt > iSCSI setup" returned nothing useful so I picked a number. If there is > someplace that actually gives detailed instructions I'm going to bookmark > it when I get the link. > > Yes, there is a storage network. There was a miscommunication and the > person attempting to create the connection didn't read instructions, so > they keyed in the main network ip, leaving a small mess for me to clean up. > The old truenas with nfs share is on the storage network and functioning > cleanly, and ovirt is able to see the iSCSI share on the storage network ip > address. > > Thank you. > > On Tue, Feb 15, 2022, 11:45 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> > wrote: > > Did you configure a storage network on oVirt ? > > About the size, why you use group '1' and not group '36' ? > > Best Regards, > Strahil Nikolov > > On Tue, Feb 15, 2022 at 20:47, David Johnson > <djohnson(a)maxistechnology.com> wrote: > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/2X7L3FUWHNPVX… > >

2 1

Problems configuring iscsi domain on ovirt 4.4.4.7-1.el8 / centos 8
by David Johnson 16 Feb '22

16 Feb '22

Since the general recommendation seems to be to use iSCSI over NFS wth ovirt, I expect that I am missing something very obvious. Question: How do I complete the iSCSI storage domain configuration? Hardware configuration: - cluster controller - physical system - compute nodes - 2 - Current file store: NFS on TrueNAS - To-be file store: iSCSI on TrueNAS Problem reproduction: 1. We decided that the problems with NFS (regularly blocking and causing VM's to go into paused instead of IOWait mode) outweighed the benefit of ease of management, so we are preparing to move to iSCSI back end on TrueNAS. 2. We created an iSCSI share on the to-be TrueNAS system and granted access to a "ovirtuser" and group "1", with 30+ TB available storage. 3. We started to add the iscsi share to the ovirt data center using the GUI, under "Storage > Storage Domains" click the "New Domain" button. *Problem 1: *Ovirt immediately found the iSCSI share (yay), but says it has 0 GB and will not allow us to add it to the data center, or edit or remove it, so it is in a limbo state. *Problem 2: *Ovirt shows it on the wrong network (miscommunication). It is finding it on the general network instead of the isolated data network. The TrueNAS will be removed from the general network as soon as configuration is complete. Thank you *David Johnson*

2 2

Cluster and datacenter compatibility levels
by Colin Coe 15 Feb '22

15 Feb '22

Hi all I'm looking for a definitive answer on what each compatibility level means. Specifically, I'm looking for: - new features - deprecated features - removed features We're on 4.3 so I'm looking for the info above on 4.4, 4.5 and 4.6 My main motivation is to ensure we don't lose SPICE support. We actually use RHV and I logged a case with Red Hat GSS but couldn't get the info I was after. Thanks

2 1

vm seal
by Nathanaël Blanchet 14 Feb '22

14 Feb '22

Hello, I noticed that a vm created from a "sealed" template is initially mount on one host with libguestfs, with a virt-sysprep process, before getting ready to be used. This should be unuseful given that the template is already sealed. Is there a reason to that? -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

2 3

Re: Backup prozess
by marcel d'heureuse 14 Feb '22

14 Feb '22

Sure: https://github.com/vacosta94/VirtBKP Br Marcel Am 14. Februar 2022 20:29:47 MEZ schrieb Jonas Liechti <jonas(a)rabe.ch>: >Would you mind sharing a link to this script? I would be interested in how it works.Am 14.02.2022 19:45 schrieb marcel d'heureuse <marcel(a)deheureu.se>: >> >> Moin, >> >> We have in our Environment 12 servers managed by one self hosted engine. It is ovirt 4.3.9. We are Freizeit on that Version. >> >> How did you make Backups? We use a github Script which generate via api a snapshot and Mount this disk into the vm where the backup Script is running. The backup Script Export this additional Hard disk to a storage and disconnect the disk and remove the snapshot. >> >> This works on Linux vm fine if they have Medium load. If we try this with a Windows 10 vm or Windows Server vm or a Linux vm with high load or with a very big Hard drive this will not work. Disk can't Export and snapshot will not delete. >> >> I have found winchin and bareos but I have not start to check this first I want to have some more possible options. >> >> Thanks >> >> Marcel

1 0

Re: Backup prozess
by Jonas Liechti 14 Feb '22

14 Feb '22

Would you mind sharing a link to this script? I would be interested in how it works.Am 14.02.2022 19:45 schrieb marcel d'heureuse <marcel(a)deheureu.se>: > > Moin, > > We have in our Environment 12 servers managed by one self hosted engine. It is ovirt 4.3.9. We are Freizeit on that Version. > > How did you make Backups? We use a github Script which generate via api a snapshot and Mount this disk into the vm where the backup Script is running. The backup Script Export this additional Hard disk to a storage and disconnect the disk and remove the snapshot. > > This works on Linux vm fine if they have Medium load. If we try this with a Windows 10 vm or Windows Server vm or a Linux vm with high load or with a very big Hard drive this will not work. Disk can't Export and snapshot will not delete. > > I have found winchin and bareos but I have not start to check this first I want to have some more possible options. > > Thanks > > Marcel

1 0

Backup prozess
by marcel d'heureuse 14 Feb '22

14 Feb '22

Moin, We have in our Environment 12 servers managed by one self hosted engine. It is ovirt 4.3.9. We are Freizeit on that Version. How did you make Backups? We use a github Script which generate via api a snapshot and Mount this disk into the vm where the backup Script is running. The backup Script Export this additional Hard disk to a storage and disconnect the disk and remove the snapshot. This works on Linux vm fine if they have Medium load. If we try this with a Windows 10 vm or Windows Server vm or a Linux vm with high load or with a very big Hard drive this will not work. Disk can't Export and snapshot will not delete. I have found winchin and bareos but I have not start to check this first I want to have some more possible options. Thanks Marcel

2 1

Remove obsolete Gluster hyperconverged doc
by Strahil Nikolov 14 Feb '22

14 Feb '22

Hi Sandro, It seems that with https://github.com/oVirt/ovirt-site/commit/f3286b2e1f2871978d054250556f1b4f… all refference to Gluster & Hyperconverged is removed from the website. Can you clarify the situation with Hyperconverged deployments ? Best Regards,Strahil Nikolov

9 12

VDSM KVM2 command SpmStatusVDS failed: Message timeout which can be caused by communication issues
by Chafik Hammami 14 Feb '22

14 Feb '22

Hello, Can you please help us to find a solution for this error VDSM KVM2 command SpmStatusVDS failed: Message timeout which can be caused by communication issues [cid:image001.jpg@01D821CF.F8705D50]

1 0

VDSM KVM2 command SpmStatusVDS failed: Message timeout which can be caused by communication issues
by c.hammami＠nextstep-it.com 14 Feb '22

14 Feb '22

can you please help us. i have 2 KVM with this error VDSM KVM2 command SpmStatusVDS failed: Message timeout which can be caused by communication issues

1 0

Re: oVirt 4.3 - Fibre Channel Data Domain as ISO dump.
by Nir Soffer 14 Feb '22

14 Feb '22

On Mon, Feb 14, 2022 at 4:31 PM Angus Clarke <angus(a)charworth.com> wrote: > > Hello > > RE: oVirt 4.3 - Fibre Channel Data Domain as ISO dump. > > Thanks for letting me join the list. Welcome! > I added a fibre channel Data domain with a view to using it as an ISO dump however I cannot mount CDs to VMs with this error: > > "Error while executing action Change CD: Drive image file could not be found" > > A bit of reading suggests there is no way around this when using fibre channel data domains - is this the case? This works since 4.4.6, but was broken in 4.3. You should upgrade to 4.4 at this point. > I guess I could present the LUN to a VM and run NFS from there as an alternative option. Yes NFS works for ISO on data domain in 4.3. Nir

1 0

oVirt 4.3 - Fibre Channel Data Domain as ISO dump.
by Angus Clarke 14 Feb '22

14 Feb '22

Hello RE: oVirt 4.3 - Fibre Channel Data Domain as ISO dump. Thanks for letting me join the list. I added a fibre channel Data domain with a view to using it as an ISO dump however I cannot mount CDs to VMs with this error: "Error while executing action Change CD: Drive image file could not be found" A bit of reading suggests there is no way around this when using fibre channel data domains - is this the case? I guess I could present the LUN to a VM and run NFS from there as an alternative option. Thanks a lot Angus

1 0

RHGS and RHV closing down: could you please put that on the home page?
by Thomas Hoberg 14 Feb '22

14 Feb '22

I just read this message: https://bugzilla.redhat.com/show_bug.cgi?id=2016359 I am shocked but not surprised. And very, very sad. But I believe this decision needs to be communicated more prominently, as people should not get aboard a project already axed.

14 27