ImageIO Performance
Hi All, We backup our VM's with a custom script based on the https://github.com/oVirt/python-ovirt-engine-sdk4/blob/main/examples/backup_... example. This works fine, but we start to see scaling issues. On VM's where there are a lot of dirty blocks, the transfer goes really slow (sometimes only 20MiB/sec). At the same time we see that ovirt-imageio process sometimes uses 100% CPU (its single threaded?). Is this known behavior? Are there ways to optimize imageio? Or any other ways to improve the download speed? Thanks Jean-Louis
On Mon, Feb 6, 2023 at 10:00 AM Jean-Louis Dupond via Users <users@ovirt.org> wrote:
Hi All,
We backup our VM's with a custom script based on the https://github.com/oVirt/python-ovirt-engine-sdk4/blob/main/examples/backup_... example. This works fine, but we start to see scaling issues.
On VM's where there are a lot of dirty blocks,
We need to see the list of extents returned by the server. The easiest way would be to enable debug logs - it will be even slower, but we will see these logs showing all extents: log.debug("Copying %s", ext) log.debug("Zeroing %s", ext) log.debug("Skipping %s", ext) It will also show other info that can help to understand why it is slow.
the transfer goes really slow (sometimes only 20MiB/sec).
Seems much slower than expected
At the same time we see that ovirt-imageio process sometimes uses 100% CPU
This is possible, it shows that you do a lot of requests.
(its single threaded?).
It uses thread per connection model. When used with backup_vm.py or other examples using the ovirt_imageio.client it usually use 4 connections per transfer so there will be 4 threads on the server size serving the data. Please share debug log of a slow backup, and info about the backup image storage for example, is this local file system or NFS? Nir
On Thu, Feb 9, 2023 at 7:03 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Feb 6, 2023 at 10:00 AM Jean-Louis Dupond via Users <users@ovirt.org> wrote: The easiest way would be to enable debug logs - it will be even slower, but we will see these logs showing all extents:
Using the --debug option Run backup_vm.py with --help to see all options.
On Thu, Feb 9, 2023 at 7:03 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, Feb 6, 2023 at 10:00 AM Jean-Louis Dupond via Users <users@ovirt.org> wrote:
Hi All,
We backup our VM's with a custom script based on the https://github.com/oVirt/python-ovirt-engine-sdk4/blob/main/examples/backup_... example. This works fine, but we start to see scaling issues.
On VM's where there are a lot of dirty blocks,
We need to see the list of extents returned by the server.
The easiest way would be to enable debug logs - it will be even slower, but we will see these logs showing all extents:
log.debug("Copying %s", ext) log.debug("Zeroing %s", ext) log.debug("Skipping %s", ext)
It will also show other info that can help to understand why it is slow.
the transfer goes really slow (sometimes only 20MiB/sec).
Seems much slower than expected
At the same time we see that ovirt-imageio process sometimes uses 100% CPU
This is possible, it shows that you do a lot of requests.
(its single threaded?).
It uses thread per connection model. When used with backup_vm.py or other examples using the ovirt_imageio.client it usually use 4 connections per transfer so there will be 4 threads on the server size serving the data.
Please share debug log of a slow backup, and info about the backup image storage for example, is this local file system or NFS?
I opened https://github.com/oVirt/ovirt-imageio/issues/175 to make debugging such issue easier. Nir
participants (2)
-
Jean-Louis Dupond -
Nir Soffer