- Users - oVirt List Archives

how to remove a failed backup operation
by Gianluca Cecchi 09 Sep '21

09 Sep '21

Hello, I was trying incremental backup with the provided /usr/share/doc/python3-ovirt-engine-sdk4/examples/backup_vm.py and began using the "full" option. But I specified an incorrect dir and during backup I got error due to filesystem full [ 156.7 ] Creating image transfer for disk '33b0f6fb-a855-465d-a628-5fce9b64496a' [ 157.8 ] Image transfer 'ccc386d3-9f9d-4727-832a-56d355d60a95' is ready --- Logging error ---, 105.02 seconds, 147.48 MiB/s Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/io.py", line 242, in _run handler.copy(req) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/io.py", line 286, in copy self._src.write_to(self._dst, req.length, self._buf) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 216, in write_to writer.write(view[:n]) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/nbd.py", line 118, in write self._client.write(self._position, buf) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/nbd.py", line 445, in write self._recv_reply(cmd) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/nbd.py", line 980, in _recv_reply if self._recv_reply_chunk(cmd): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/nbd.py", line 1031, in _recv_reply_chunk self._handle_error_chunk(length, flags) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/nbd.py", line 1144, in _handle_error_chunk raise ReplyError(code, message) ovirt_imageio._internal.nbd.ReplyError: Writing to file failed: [Error 28] No space left on device Now if I try the same backup command (so with "full" option) and I get ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot backup VM. The VM is during a backup operation.]". HTTP response code is 409. How can I clean the situation? BTW: the parameter to put into ovirt.conf is backup-dir or backup_dir or what? Thanks, Gianluca

4 9

Re: Poor gluster performances over 10Gbps network
by Mathieu Valois 09 Sep '21

09 Sep '21

Default shard size was 64MB. I've tested to create another volume with a 512MB shard size and settings you provided. I observe a performance improvement with `dd`, which does a sequential write (from 1MBps to 50MBps). However, I find VM still slow : as an example, `apt upgrade` is very slow to unpack packages, more than 1 hour for around 400MB (90 packages). Le 08/09/2021 à 11:34, Alex K a écrit : > > > On Wed, Sep 8, 2021 at 12:33 PM Alex K <rightkicktech(a)gmail.com > <mailto:rightkicktech@gmail.com>> wrote: > > > > On Wed, Sep 8, 2021 at 12:15 PM Mathieu Valois <mvalois(a)teicee.com > <mailto:mvalois@teicee.com>> wrote: > > Sorry for double post but I don't know if this mail has been > received. > > Hello everyone, > > I know this issue was already treated on this mailing list. > However none of the proposed solutions is satisfying me. > > Here is my situation : I've got 3 hyperconverged gluster ovirt > nodes, with 6 network interfaces, bounded in bunches of 2 > (management, VMs and gluster). The gluster network is on a > dedicated bound where the 2 interfaces are directly connected > to the 2 other ovirt nodes. Gluster is apparently using it : > > # gluster volume status vmstore > Status of volume: vmstore > Gluster process TCP Port RDMA > Port Online Pid > ------------------------------------------------------------------------------ > Brick gluster-ov1:/gluster_bricks > /vmstore/vmstore 49152 0 Y 3019 > Brick gluster-ov2:/gluster_bricks > /vmstore/vmstore 49152 0 Y 3009 > Brick gluster-ov3:/gluster_bricks > /vmstore/vmstore > > where 'gluster-ov{1,2,3}' are domain names referencing nodes > in the gluster network. This networks has 10Gbps capabilities : > > # iperf3 -c gluster-ov3 > Connecting to host gluster-ov3, port 5201 > [ 5] local 10.20.0.50 port 46220 connected to 10.20.0.51 > port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 1.16 GBytes 9.92 Gbits/sec > 17 900 KBytes > [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec > 0 900 KBytes > [ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec > 4 996 KBytes > [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec > 1 996 KBytes > [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec > 0 996 KBytes > [ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec > 0 996 KBytes > [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec > 0 996 KBytes > [ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec > 0 996 KBytes > [ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec > 0 996 KBytes > [ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec > 0 996 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec > 22 sender > [ 5] 0.00-10.04 sec 11.5 GBytes 9.86 > Gbits/sec receiver > > iperf Done. > > However, VMs stored on the vmstore gluster volume has poor > write performances, oscillating between 100KBps and 30MBps. I > almost always observe a write spike (180Mbps) at the beginning > until around 500MB written, then it drastically falls at > 10MBps, sometimes even less (100KBps). Hypervisors have 32 > threads (2 sockets, 8 cores per socket, 2 threads per core). > > Here is the volume settings : > > Volume Name: vmstore > Type: Replicate > Volume ID: XXX > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: gluster-ov1:/gluster_bricks/vmstore/vmstore > Brick2: gluster-ov2:/gluster_bricks/vmstore/vmstore > Brick3: gluster-ov3:/gluster_bricks/vmstore/vmstore > Options Reconfigured: > performance.io-thread-count: 32 # was 16 by default. > cluster.granular-entry-heal: enable > storage.owner-gid: 36 > storage.owner-uid: 36 > cluster.lookup-optimize: off > server.keepalive-count: 5 > server.keepalive-interval: 2 > server.keepalive-time: 10 > server.tcp-user-timeout: 20 > network.ping-timeout: 30 > server.event-threads: 4 > client.event-threads: 8 # was 4 by default > cluster.choose-local: off > features.shard: on > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > performance.strict-o-direct: on > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > auth.allow: * > user.cifs: off > storage.fips-mode-rchecksum: on > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: on > > I see you have not enabled sharding for the gluster volume that is > hosting VMs. I usually set a shard size of 512MB. Note though that > changing shard size in a live system will corrupt your data. > You need to wipe and set again. > > I mean you have not changed the default sharde size which is I think 4MB. > > > I've seen also good perf with the following tweaks: > > gluster volume set vms remote-dio enable > > gluster volume set vms performance.write-behind-window-size 8MB > > gluster volume set vms performance.cache-size 1GB > > gluster volume set vms performance.io-thread-count 16 > > gluster volume set vms performance.readdir-ahead on > > gluster volume set vms client.event-threads 8 > > gluster volume set vms server.event-threads 8 > > gluster volume set engine features.shard-block-size 512MB > > When I naively write directly on the logical volume, which is > mounted on a material RAID5 3-disks array, I have interesting > performances: > > # dd if=/dev/zero of=a bs=4M count=2048 > 2048+0 records in > 2048+0 records out > 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 17.2485 s, 498 > MB/s #urandom gives around 200MBps > > Moreover, hypervisors have SSD which have been configured as > lvcache, but I'm unsure how to test it efficiently. > > I can't find where is the problem, as every piece of the chain > is apparently doing well ... > > Thanks anyone for helping me :) > > -- > téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu > Valois* > > Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 > Bretteville-sur-Odon > Bureau Vitré: Zone de la baratière - 12, route de Domalain - > 35500 Vitré > 02 72 34 13 20 | www.teicee.com > <https://www.teicee.com/?pk_campaign=Email> > > téïcée sur facebook <https://www.facebook.com/teicee> téïcée > sur twitter <https://twitter.com/Teicee_fr> téïcée sur > linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur > viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org > <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > <https://www.ovirt.org/privacy-policy.html> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/BTY7TJLMQUJ3Q… > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/BTY7TJLMQUJ3Q…> > -- téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois* Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email> téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter <https://twitter.com/Teicee_fr> téïcée sur linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké

1 0

data storage domain iso upload problem
by csabany＠freemail.hu 09 Sep '21

09 Sep '21

Hi, I managed a ovirt 4.4.7 for production systems. Last week i removed the Master storage domain (moved tamplates, vm-s well, unattached, etc), but i forgot to move isos. Now, when i upload a new iso to a data storage domain, the system show it, but it's unbootable: "could not read from cdrom code 0005" thanks csabany

4 6

Re: Poor gluster performances over 10Gbps network
by Strahil Nikolov 09 Sep '21

09 Sep '21

Did you check https://lists.ovirt.org/archives/list/users@ovirt.org/message/S7I3PQVERQZT6… ? Best Regards,Strahil Nikolov On Wed, Sep 8, 2021 at 14:25, Staniforth, Paul<P.Staniforth(a)leedsbeckett.ac.uk> wrote: _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DGITVP5DCEF33…

1 0

Re: Time Drift Issues
by Marcos Sungaila 08 Sep '21

08 Sep '21

Hi Nur, Yes, oVirt Node comes with an NTP client installed and started. You need to configure the servers it will use as external sources. Yes, it will be preserved in a host upgrade. Regards, Marcos From: Nur Imam Febrianto <nur_imam(a)outlook.com> Sent: quarta-feira, 8 de setembro de 2021 08:09 To: Marcos Sungaila <marcos.sungaila(a)oracle.com>; oVirt Users <users(a)ovirt.org> Subject: [External] : RE: Time Drift Issues Hi Marcos, Want to clarify one thing. If I'm using and oVirt Node based host (not EL Linux Based), does it already configured with ntp client or can I just configure any ntp client on the host ? If I just configure ntp client on the host either chronyc or ntpd, will it persist after upgrading the node ? Thanks before. Regards, Nur Imam Febrianto Sent from Mail<https://urldefense.com/v3/__https:/go.microsoft.com/fwlink/?LinkId=550986__…> for Windows From: Marcos Sungaila<mailto:marcos.sungaila@oracle.com> Sent: Tuesday, 07 September 2021 20:54 To: Nur Imam Febrianto<mailto:nur_imam@outlook.com>; oVirt Users<mailto:users@ovirt.org> Subject: RE: Time Drift Issues Hi Nur, ntpd has a 300sec limit for synchronization. If you want a more flexible NTP client, use chronyd. If you prefer to use ntpd, you should have ntpdate as a boot client and ntpd as a runtime client. When the server boots, the ntpdate client will sync time no matter the difference. As a boot client, it runs only once and exits. Then the ntpd service starts, and keep your server clock in sync with an external source. On distributions like CentOS and similar, ntpdate reads the external time sources configuration from ntpd.conf. You should need booth packages and services installed and enabled: I prefer chronyd since it does not need any extra service or procedure to keep clock synchronization. Regards, Marcos From: Nur Imam Febrianto <nur_imam(a)outlook.com<mailto:nur_imam@outlook.com>> Sent: terça-feira, 7 de setembro de 2021 01:08 To: oVirt Users <users(a)ovirt.org<mailto:users@ovirt.org>> Subject: [External] : [ovirt-users] Time Drift Issues Hi All, Recently I got an warning in our cluster about time drift : Host xxxxx has time-drift of 672 seconds while maximum configured value is 300 seconds. What should I do to address this issue ? Should I reconfigure / configure ntp client on all host ? Thanks before. Regards, Nur Imam Febrianto Sent from Mail<https://urldefense.com/v3/__https:/apac01.safelinks.protection.outlook.com/…> for Windows

1 0

Re: [External] : Re: Poor gluster performances over 10Gbps network
by Marcos Sungaila 08 Sep '21

08 Sep '21

Hi Mathieu, Using dd for disk performance tests is not recommended. Besides needing oflag and iflag options, dd will perform a sequential write what is not a real scenario, To evaluate your servers, use fio or bonnie++ for performance tests. These applications will play a more realistic scenario to validate your environment. Regards, Marcos From: Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk> Sent: quarta-feira, 8 de setembro de 2021 08:17 To: Mathieu Valois <mvalois(a)teicee.com>; users <users(a)ovirt.org> Subject: [External] : [ovirt-users] Re: Poor gluster performances over 10Gbps network Hi Mathieu, with a linux VM using dd without the oflag=sync will mean it's using disk bufffers, hence the faster thoughput at the beginning until the buffers are full. Regards, Paul S. ________________________________ From: Mathieu Valois <mvalois(a)teicee.com<mailto:mvalois@teicee.com>> Sent: 08 September 2021 11:56 To: Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk<mailto:P.Staniforth@leedsbeckett.ac.uk>>; users <users(a)ovirt.org<mailto:users@ovirt.org>> Subject: Re: [ovirt-users] Poor gluster performances over 10Gbps network Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Hi Paul, thank you for your answer. Indeed I did a `dd` inside a VM to measure the Gluster disk performance. I've also tried `dd` on an hypervisor writing into one of the replicated gluster bricks, which gives good performances (similar to logical volume ones). Le 08/09/2021 à 12:51, Staniforth, Paul a écrit : Hi Mathieu, How are you measuring the Gluster disk performance? also using dd you should use the oflag=dsync to avoid buffer caching. Regards, Paul S ________________________________ From: Mathieu Valois <mvalois(a)teicee.com><mailto:mvalois@teicee.com> Sent: 08 September 2021 10:12 To: users <users(a)ovirt.org><mailto:users@ovirt.org> Subject: [ovirt-users] Poor gluster performances over 10Gbps network Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Sorry for double post but I don't know if this mail has been received. Hello everyone, I know this issue was already treated on this mailing list. However none of the proposed solutions is satisfying me. Here is my situation : I've got 3 hyperconverged gluster ovirt nodes, with 6 network interfaces, bounded in bunches of 2 (management, VMs and gluster). The gluster network is on a dedicated bound where the 2 interfaces are directly connected to the 2 other ovirt nodes. Gluster is apparently using it : # gluster volume status vmstore Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gluster-ov1:/gluster_bricks /vmstore/vmstore 49152 0 Y 3019 Brick gluster-ov2:/gluster_bricks /vmstore/vmstore 49152 0 Y 3009 Brick gluster-ov3:/gluster_bricks /vmstore/vmstore where 'gluster-ov{1,2,3}' are domain names referencing nodes in the gluster network. This networks has 10Gbps capabilities : # iperf3 -c gluster-ov3 Connecting to host gluster-ov3, port 5201 [ 5] local 10.20.0.50 port 46220 connected to 10.20.0.51 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.16 GBytes 9.92 Gbits/sec 17 900 KBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 900 KBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 4 996 KBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 1 996 KBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 996 KBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec 0 996 KBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 22 sender [ 5] 0.00-10.04 sec 11.5 GBytes 9.86 Gbits/sec receiver iperf Done. However, VMs stored on the vmstore gluster volume has poor write performances, oscillating between 100KBps and 30MBps. I almost always observe a write spike (180Mbps) at the beginning until around 500MB written, then it drastically falls at 10MBps, sometimes even less (100KBps). Hypervisors have 32 threads (2 sockets, 8 cores per socket, 2 threads per core). Here is the volume settings : Volume Name: vmstore Type: Replicate Volume ID: XXX Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster-ov1:/gluster_bricks/vmstore/vmstore Brick2: gluster-ov2:/gluster_bricks/vmstore/vmstore Brick3: gluster-ov3:/gluster_bricks/vmstore/vmstore Options Reconfigured: performance.io-thread-count: 32 # was 16 by default. cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 cluster.lookup-optimize: off server.keepalive-count: 5 server.keepalive-interval: 2 server.keepalive-time: 10 server.tcp-user-timeout: 20 network.ping-timeout: 30 server.event-threads: 4 client.event-threads: 8 # was 4 by default cluster.choose-local: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable performance.strict-o-direct: on network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: on When I naively write directly on the logical volume, which is mounted on a material RAID5 3-disks array, I have interesting performances: # dd if=/dev/zero of=a bs=4M count=2048 2048+0 records in 2048+0 records out 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 17.2485 s, 498 MB/s #urandom gives around 200MBps Moreover, hypervisors have SSD which have been configured as lvcache, but I'm unsure how to test it efficiently. I can't find where is the problem, as every piece of the chain is apparently doing well ... Thanks anyone for helping me :) -- [téïcée]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> Mathieu Valois Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> [téïcée sur facebook]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…>[téïcée sur twitter]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…>[téïcée sur linkedin]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…>[téïcée sur viadeo]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> [Datadocké] To view the terms under which this email is distributed, please go to:- https://leedsbeckett.ac.uk/disclaimer/email<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> -- [téïcée]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> Mathieu Valois Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> [téïcée sur facebook]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…>[téïcée sur twitter]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…>[téïcée sur linkedin]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…>[téïcée sur viadeo]<https://urldefense.com/v3/__https:/eur02.safelinks.protection.outlook.com/?…> [Datadocké] To view the terms under which this email is distributed, please go to:- https://leedsbeckett.ac.uk/disclaimer/email<https://urldefense.com/v3/__https:/leedsbeckett.ac.uk/disclaimer/email__;!!…>

1 0

oVirt 2021 Conference is almost here!
by Sandro Bonazzola 08 Sep '21

08 Sep '21

Hello oVirt friends, The oVirt 2021 Conference is almost here! The videos are uploaded, live speakers tech-checked, and last but not least, the final schedule with the watch links is available on the oVirt website <https://blogs.ovirt.org/ovirt-2021-online-conference/>[1]. Most talks are pre-recorded, but the speakers will be in the chat and answer your questions during their talk. See you all there, The organizers [1] https://blogs.ovirt.org/ovirt-2021-online-conference/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: Time Drift Issues
by Gianluca Cecchi 08 Sep '21

08 Sep '21

On Wed, Sep 8, 2021 at 1:19 PM Nur Imam Febrianto <nur_imam(a)outlook.com> wrote: > Hi Marcos, > > > > Want to clarify one thing. If I’m using and oVirt Node based host (not EL > Linux Based), does it already configured with ntp client or can I just > configure any ntp client on the host ? If I just configure ntp client on > the host either chronyc or ntpd, will it persist after upgrading the node ? > > > > Thanks before. > > > > Regards, > > Nur Imam Febrianto > > > > Just to clarify: when using oVirt node 4.4, during installation you have the usual way to configure ntp inside anaconda. In 4.4, based on CentOS (stream) 8, when done that way it will configure chronyd. In the recent past there was a bug where sometimes installation crashed when configuring ntp in certain conditions. If you don't configure ntp at install time you can configure it later by modifying /etc/chrony.conf and starting/enabling the service from the cockpit web interface (or command line with usual commands). It will remain across reboots and/or image updates. HIH, Gianluca

1 0

Re: Poor gluster performances over 10Gbps network
by Mathieu Valois 08 Sep '21

08 Sep '21

Hi Paul, thank you for your answer. Indeed I did a `dd` inside a VM to measure the Gluster disk performance. I've also tried `dd` on an hypervisor writing into one of the replicated gluster bricks, which gives good performances (similar to logical volume ones). Le 08/09/2021 à 12:51, Staniforth, Paul a écrit : > > Hi Mathieu, > How are you measuring the Gluster disk performance? > also using dd you should use the > oflag=dsync > > to avoid buffer caching. > > > Regards, > > Paul S > ------------------------------------------------------------------------ > *From:* Mathieu Valois <mvalois(a)teicee.com> > *Sent:* 08 September 2021 10:12 > *To:* users <users(a)ovirt.org> > *Subject:* [ovirt-users] Poor gluster performances over 10Gbps network > > *Caution External Mail:* Do not click any links or open any > attachments unless you trust the sender and know that the content is safe. > > Sorry for double post but I don't know if this mail has been received. > > Hello everyone, > > I know this issue was already treated on this mailing list. However > none of the proposed solutions is satisfying me. > > Here is my situation : I've got 3 hyperconverged gluster ovirt nodes, > with 6 network interfaces, bounded in bunches of 2 (management, VMs > and gluster). The gluster network is on a dedicated bound where the 2 > interfaces are directly connected to the 2 other ovirt nodes. Gluster > is apparently using it : > > # gluster volume status vmstore > Status of volume: vmstore > Gluster process TCP Port RDMA Port > Online Pid > ------------------------------------------------------------------------------ > Brick gluster-ov1:/gluster_bricks > /vmstore/vmstore 49152 0 > Y 3019 > Brick gluster-ov2:/gluster_bricks > /vmstore/vmstore 49152 0 > Y 3009 > Brick gluster-ov3:/gluster_bricks > /vmstore/vmstore > > where 'gluster-ov{1,2,3}' are domain names referencing nodes in the > gluster network. This networks has 10Gbps capabilities : > > # iperf3 -c gluster-ov3 > Connecting to host gluster-ov3, port 5201 > [ 5] local 10.20.0.50 port 46220 connected to 10.20.0.51 port 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 1.16 GBytes 9.92 Gbits/sec 17 900 KBytes > [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 900 KBytes > [ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 4 996 KBytes > [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 1 996 KBytes > [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 996 KBytes > [ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes > [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes > [ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec 0 996 KBytes > [ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes > [ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec > 22 sender > [ 5] 0.00-10.04 sec 11.5 GBytes 9.86 > Gbits/sec receiver > > iperf Done. > > However, VMs stored on the vmstore gluster volume has poor write > performances, oscillating between 100KBps and 30MBps. I almost always > observe a write spike (180Mbps) at the beginning until around 500MB > written, then it drastically falls at 10MBps, sometimes even less > (100KBps). Hypervisors have 32 threads (2 sockets, 8 cores per socket, > 2 threads per core). > > Here is the volume settings : > > Volume Name: vmstore > Type: Replicate > Volume ID: XXX > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: gluster-ov1:/gluster_bricks/vmstore/vmstore > Brick2: gluster-ov2:/gluster_bricks/vmstore/vmstore > Brick3: gluster-ov3:/gluster_bricks/vmstore/vmstore > Options Reconfigured: > performance.io-thread-count: 32 # was 16 by default. > cluster.granular-entry-heal: enable > storage.owner-gid: 36 > storage.owner-uid: 36 > cluster.lookup-optimize: off > server.keepalive-count: 5 > server.keepalive-interval: 2 > server.keepalive-time: 10 > server.tcp-user-timeout: 20 > network.ping-timeout: 30 > server.event-threads: 4 > client.event-threads: 8 # was 4 by default > cluster.choose-local: off > features.shard: on > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > performance.strict-o-direct: on > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > auth.allow: * > user.cifs: off > storage.fips-mode-rchecksum: on > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: on > > When I naively write directly on the logical volume, which is mounted > on a material RAID5 3-disks array, I have interesting performances: > > # dd if=/dev/zero of=a bs=4M count=2048 > 2048+0 records in > 2048+0 records out > 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 17.2485 s, 498 MB/s > #urandom gives around 200MBps > > Moreover, hypervisors have SSD which have been configured as lvcache, > but I'm unsure how to test it efficiently. > > I can't find where is the problem, as every piece of the chain is > apparently doing well ... > > Thanks anyone for helping me :) > > -- > téïcée > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.teice…> > *Mathieu Valois* > > Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 > Bretteville-sur-Odon > Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré > 02 72 34 13 20 | www.teicee.com > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.teice…> > > > téïcée sur facebook > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.faceb…> > téïcée sur twitter > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.c…> > téïcée sur linkedin > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linke…> > téïcée sur viadeo > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffr.viadeo…> > Datadocké > > > To view the terms under which this email is distributed, please go to:- > https://leedsbeckett.ac.uk/disclaimer/email > <https://leedsbeckett.ac.uk/disclaimer/email> -- téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois* Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email> téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter <https://twitter.com/Teicee_fr> téïcée sur linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké

2 1

Re: Time Drift Issues
by Nur Imam Febrianto 08 Sep '21

08 Sep '21

Hi Marcos, Want to clarify one thing. If I’m using and oVirt Node based host (not EL Linux Based), does it already configured with ntp client or can I just configure any ntp client on the host ? If I just configure ntp client on the host either chronyc or ntpd, will it persist after upgrading the node ? Thanks before. Regards, Nur Imam Febrianto Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows From: Marcos Sungaila<mailto:marcos.sungaila@oracle.com> Sent: Tuesday, 07 September 2021 20:54 To: Nur Imam Febrianto<mailto:nur_imam@outlook.com>; oVirt Users<mailto:users@ovirt.org> Subject: RE: Time Drift Issues Hi Nur, ntpd has a 300sec limit for synchronization. If you want a more flexible NTP client, use chronyd. If you prefer to use ntpd, you should have ntpdate as a boot client and ntpd as a runtime client. When the server boots, the ntpdate client will sync time no matter the difference. As a boot client, it runs only once and exits. Then the ntpd service starts, and keep your server clock in sync with an external source. On distributions like CentOS and similar, ntpdate reads the external time sources configuration from ntpd.conf. You should need booth packages and services installed and enabled: I prefer chronyd since it does not need any extra service or procedure to keep clock synchronization. Regards, Marcos From: Nur Imam Febrianto <nur_imam(a)outlook.com> Sent: terça-feira, 7 de setembro de 2021 01:08 To: oVirt Users <users(a)ovirt.org> Subject: [External] : [ovirt-users] Time Drift Issues Hi All, Recently I got an warning in our cluster about time drift : Host xxxxx has time-drift of 672 seconds while maximum configured value is 300 seconds. What should I do to address this issue ? Should I reconfigure / configure ntp client on all host ? Thanks before. Regards, Nur Imam Febrianto Sent from Mail<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefen…> for Windows

1 0