2:52 p.m.
--jigfid2yHjNFZUTO
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 05/25 14:42, Barak Korren wrote:
On 25 May 2016 at 12:44, Eyal Edri <eedri(a)redhat.com> wrote:
> OK,
> I suggest to test using a VM with local disk (preferably on a host with=
SSD
> configured), if its working,
> lets expedite moving all VMs or at least a large amount of VMs to it un=
til
> we see network load reduced.
>
=20
This is not that easy, oVirt doesn't support mixing local disk and
storage in the same cluster, so we will need to move hosts to a new
cluster for this.
Also we will lose the ability to use templates, or otherwise have to
create the templates on each and every disk.
=20
The scratch disk is a good solution for this, where you can have the
OS image on the central storage and the ephemeral data on the local
disk.
=20
WRT to the storage architecture - a single huge (10.9T) ext4 is used
as the FS on top of the DRBD, this is probably not the most efficient
thing one can do (XFS would probably have been better, RAW via iSCSI -
even better).
That was done >3 years ago, xfs was not quite stable and widely used and
supported back then.
=20
I'm guessing that those 10/9TB are not made from a single disk but
with a hardware RAID of some sort. In this case deactivating the
hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That
are then re-joined to a single sotrage domain in oVirt) will enable
different VMs to concurrently work on different disks. This should
lower the per-vm storage latency.
That would get rid of the drbd too, it's a totally different setup, from
scratch (no nfs either).
=20
Looking at the storage machine I see strong indication it is IO bound
- the load average is ~12 while there are just 1-5 working processes
and the CPU is ~80% idle and the rest is IO wait.
=20
Running 'du *' at:
/srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images
one can see that most images are ~40G in size (that is _real_ 40G not
sparse!). This means that despite having most VMs created based on
templates, the VMs are full template copies rather then COW clones.
That should not be like that, maybe the templates are wrongly configured? or
foreman images?
What this means is that using pools (where all VMs are COW copies of
the single pool template) is expected to significantly reduce the
storage utilization and therefore the IO load on it (the less you
store, the less you need to read back).
That should happen too without pools, with normal qcow templates.
And in any case, that will not lower the normal io, when not actually creat=
ing
vms, as any read and write will still hit the disk anyhow, it only alleviat=
es
the io when creating new vms. The local disk (scratch disk) is the best opt=
ion
imo, now and for the foreseeable future.
=20
--=20
Barak Korren
bkorren(a)redhat.com
RHEV-CI Team
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--=20
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: dcaro(a)redhat.com
IRC: dcaro|dcaroest@{freenode|oftc|redhat}
Web: www.redhat.com
RHT Global #: 82-62605
--jigfid2yHjNFZUTO
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJXRZIOAAoJEEBxx+HSYmnDtxoH/Rr9aRoqydoTprHsg7VEwNx0
ZgsGevCm8NIx5BFRgCt+exllDFQx9hv6D6346NPPotLl/z5WP6NY6GZWguKzG2pO
IWztPlSubZJS5dpqJqwCnMn3jDjFB1XgCCcPd6hRYPFYe5lP1qJm6qPuH+R9zpWD
uxZ5iNkUf+tt0JFsb/1/1DthVWU1shGWVRU+P2XijUlg5eZQFtAc5x7eKu8UFDxh
gt+zthA3MHkeadiqHyiIWTHDmDzM9Us7B3U9dBzdSp9g5Jmo9qPh+FqrU6oH9cdI
Szhz+l8OTlLGPVh9an8XbNQMUSy1UDvBZB4GZ9qQbosHyPI5zZNbky0csuGdqMU=
=jejh
-----END PGP SIGNATURE-----
--jigfid2yHjNFZUTO--