On 05/24 17:57, Fabian Deutsch wrote: > Hey, >=20 > $subj says it all. >=20 > Affected jobs are: > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ >=20 > I.e. 3.6 - before: ~46min, now 1:23hrs >=20 > In master it's even worse: >1:30hrs >=20 > Can someone help to idnetify the reason? =20 =20 I see that this is where there's a big jump in time: =20 06:39:38 Domain installation still in progress. You can reconnect to=20 06:39:38 the console to complete the installation process. 07:21:51 ................................................................=

=20 So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to w=

=20 >=20 > - fabian >=20 > --=20 > Fabian Deutsch <fdeutsch(a)redhat.com&gt; > RHEV Hypervisor > Red Hat > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra =20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRHwPAAoJEEBxx+HSYmnD568IAIgydp+kOT7lxkmqGxT8fR1Q X4I7RBHDCrZwnXmtaeebxtV3U7EwkorIEQiKXJVgqwJM5qtqeLTQUXNd8MU/sco5 Hz4jLuydM1W7amVPfj6ka4ZBemy6pghNF+3zRdUgMFQwFwMsQ9/anAZK/sAcO0/M +NToKWNbG/cCVj3Dx24ATDUYvXjDxn27yp9b7e0etswliO+/0Jb9XZOwnZwU8SKu hXIy1x5e/3c8pwMrli/TI5vPxns5A9ubEEFmlWzczhHz7y/RrdcxkNmfNHLGeawx nxpq1Kidxh8NUqxoZ41P/HVO6Bm7Ir5T2JOs+XhKii1eLvH13m6EMWCmpeNnSwY= =OtiM -----END PGP SIGNATURE-----

Is it running on the same slave as before? Can we see a changelog of things changed from the Last time it took less time? On May 24, 2016 7:06 PM, "David Caro" <dcaro(a)redhat.com&gt; wrote: > > On 05/24 18:03, David Caro wrote: > > On 05/24 17:57, Fabian Deutsch wrote: > > > Hey, > > > > > > $subj says it all. > > > > > > Affected jobs are: > > > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ > > > > > > I.e. 3.6 - before: ~46min, now 1:23hrs > > > > > > In master it's even worse: >1:30hrs > > > > > > Can someone help to idnetify the reason? > > > > > > I see that this is where there's a big jump in time: > > > > 06:39:38 Domain installation still in progress. You can reconnect to > > 06:39:38 the console to complete the installation process. > > 07:21:51 > > .............................................................................................................................................................................................................................................................2016-05-24 > > 03:21:51,341: Install finished. Or at least virt shut down. > > > > So it looks as if the code that checks if the domain is shut down is not > > working properly, or maybe the virt-install is taking very very long to > > work. > > > It seems that the virt-install log is not being archived, and the workdir > has > already been cleaned up so I can check the logfile: > > > /home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log > > > Maybe you can archive it too on the next run to debug > > > > > > > > > - fabian > > > > > > -- > > > Fabian Deutsch <fdeutsch(a)redhat.com&gt; > > > RHEV Hypervisor > > > Red Hat > > > _______________________________________________ > > > Infra mailing list > > > Infra(a)ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/infra > > > > -- > > David Caro > > > > Red Hat S.L. > > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > > > Tel.: +420 532 294 605 > > Email: dcaro(a)redhat.com > > IRC: dcaro|dcaroest@{freenode|oftc|redhat} > > Web: www.redhat.com > > RHT Global #: 82-62605 > > > > -- > David Caro > > Red Hat S.L. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Tel.: +420 532 294 605 > Email: dcaro(a)redhat.com > IRC: dcaro|dcaroest@{freenode|oftc|redhat} > Web: www.redhat.com > RHT Global #: 82-62605 > > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra >

btw, feel free to increase history for node builds now, we have the space for it. I wouldn't got crazy with keeping history of artifacts, but build info with console output there is no problem keeping a month old.

I'm adding Barak as this weekly infra owner to help debug this tomorrow if needed. Also, Evgheni is working on network configuration, might be relevant as well.

On Tue, May 24, 2016 at 9:30 PM, Fabian Deutsch <fdeutsch(a)redhat.com&gt; wrote: > > I don't know if it was quicker on different salves, because I can not > look at the history. > > There were no Node sided changes, maybe the package set in master got > larger - well see that on the first regular build. > > But: I enabled kvm in mock, and this seems to help. The first job post > this chnage looks good. > > My assumption is that previous builds were running on a beefier slave. > > - fabian > > On Tue, May 24, 2016 at 6:50 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > > Is it running on the same slave as before? Can we see a changelog of > > things > > changed from the Last time it took less time? > > > > On May 24, 2016 7:06 PM, "David Caro" <dcaro(a)redhat.com&gt; wrote: > >> > >> On 05/24 18:03, David Caro wrote: > >> > On 05/24 17:57, Fabian Deutsch wrote: > >> > > Hey, > >> > > > >> > > $subj says it all. > >> > > > >> > > Affected jobs are: > >> > > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ > >> > > > >> > > I.e. 3.6 - before: ~46min, now 1:23hrs > >> > > > >> > > In master it's even worse: >1:30hrs > >> > > > >> > > Can someone help to idnetify the reason? > >> > > >> > > >> > I see that this is where there's a big jump in time: > >> > > >> > 06:39:38 Domain installation still in progress. You can reconnect to > >> > 06:39:38 the console to complete the installation process. > >> > 07:21:51 > >> > > >> > .............................................................................................................................................................................................................................................................2016-05-24 > >> > 03:21:51,341: Install finished. Or at least virt shut down. > >> > > >> > So it looks as if the code that checks if the domain is shut down is > >> > not > >> > working properly, or maybe the virt-install is taking very very long > >> > to > >> > work. > >> > >> > >> It seems that the virt-install log is not being archived, and the > >> workdir > >> has > >> already been cleaned up so I can check the logfile: > >> > >> > >> > >> /home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log > >> > >> > >> Maybe you can archive it too on the next run to debug > >> > >> > > >> > > > >> > > - fabian > >> > > > >> > > -- > >> > > Fabian Deutsch <fdeutsch(a)redhat.com&gt; > >> > > RHEV Hypervisor > >> > > Red Hat > >> > > _______________________________________________ > >> > > Infra mailing list > >> > > Infra(a)ovirt.org > >> > > http://lists.ovirt.org/mailman/listinfo/infra > >> > > >> > -- > >> > David Caro > >> > > >> > Red Hat S.L. > >> > Continuous Integration Engineer - EMEA ENG Virtualization R&D > >> > > >> > Tel.: +420 532 294 605 > >> > Email: dcaro(a)redhat.com > >> > IRC: dcaro|dcaroest@{freenode|oftc|redhat} > >> > Web: www.redhat.com > >> > RHT Global #: 82-62605 > >> > >> > >> > >> -- > >> David Caro > >> > >> Red Hat S.L. > >> Continuous Integration Engineer - EMEA ENG Virtualization R&D > >> > >> Tel.: +420 532 294 605 > >> Email: dcaro(a)redhat.com > >> IRC: dcaro|dcaroest@{freenode|oftc|redhat} > >> Web: www.redhat.com > >> RHT Global #: 82-62605 > >> > >> _______________________________________________ > >> Infra mailing list > >> Infra(a)ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/infra > >> > > > > > > -- > Fabian Deutsch <fdeutsch(a)redhat.com&gt; > RHEV Hypervisor > Red Hat -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch(a)redhat.com&gt; ha scritto: > > Hey, > > $subj says it all. > > Affected jobs are: > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ > > I.e. 3.6 - before: ~46min, now 1:23hrs > > In master it's even worse: >1:30hrs > > Can someone help to idnetify the reason? I have no numbers but I have the feeling that all jobs are getting slower since a couple of weeks ago. Yum install phase takes ages. I thoughtit was some temporary storage i/o peak but looks like it's not temporary. > > - fabian > > -- > Fabian Deutsch <fdeutsch(a)redhat.com&gt; > RHEV Hypervisor > Red Hat > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

Fabian, Please open a ticket to enable all VMs to support nested, I don't see a reason why not to. Evgheni - we'll need to make sure all hosts supports nested, it will require reboot of a server if its not enabled now.

If that is the case, and the required need can be set in the automation dir the I agree. Is that possible to set via scripts in the automation dir?

Hi, I checked SAR data on storage servers and compared loads yesterday and three weeks ago (May 3). Load values are in pretty much the same range, yet now most of the time they are around the "high" mark so we may be nearing a bottleneck, specifically on I/O where we mostly do writes to the NAS, not reads and there's quite a bit of overhead: VM -> QCOW -> file -> network -> NFS -> DRBD -> disk Surely, using local scratch disks stored on SSDs should greatly improve performance as at least half of the above steps will be gone. We don't really need to centrally store (NFS) or mirror (DRBD) data that slaves write to their disks all the time anyways. For VMs where we do need redundancy, I'd suggest using iSCSI storage domains in the long run. Regards, Evgheni Dereveanchin ----- Original Message ----- From: "Eyal Edri" <eedri(a)redhat.com&gt; To: "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt;, "Evgheni Dereveanchin" < ederevea(a)redhat.com&gt;, "Anton Marchukov" <amarchuk(a)redhat.com&gt; Cc: "Fabian Deutsch" <fdeutsch(a)redhat.com&gt;, "infra" <infra(a)ovirt.org&gt; Sent: Wednesday, 25 May, 2016 9:31:43 AM Subject: Re: ngn build jobs take more than twice (x) as long as in the last days It might be more load on the storage servers with now running much more jobs. Evgheni - can you check if the load on the storage servers has changed significantly to justify this degradation of service? We need to expedite the enablement of SSDs in the hypervisors and move to local hooks. Anton - do we have a test VM that uses a local DISK we can use to test if it improves the runtime? On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch(a)redhat.com&gt; ha scritto: > > > > Hey, > > > > $subj says it all. > > > > Affected jobs are: > > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ > > > > I.e. 3.6 - before: ~46min, now 1:23hrs > > > > In master it's even worse: >1:30hrs > > > > Can someone help to idnetify the reason? > > I have no numbers but I have the feeling that all jobs are getting slower > since a couple of weeks ago. Yum install phase takes ages. I thoughtit was > some temporary storage i/o peak but looks like it's not temporary. > > > > > - fabian > > > > -- > > Fabian Deutsch <fdeutsch(a)redhat.com&gt; > > RHEV Hypervisor > > Red Hat > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

On 25 May 2016 at 12:44, Eyal Edri <eedri(a)redhat.com&gt; wrote: > OK, > I suggest to test using a VM with local disk (preferably on a host with=

> configured), if its working, > lets expedite moving all VMs or at least a large amount of VMs to it un=

> we see network load reduced. > =20 This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk. =20 The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk. =20 WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI - even better).

=20 I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency.

=20 Looking at the storage machine I see strong indication it is IO bound - the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait. =20 Running 'du *' at: /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones.

What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back).

=20 --=20 Barak Korren bkorren(a)redhat.com RHEV-CI Team _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRZIOAAoJEEBxx+HSYmnDtxoH/Rr9aRoqydoTprHsg7VEwNx0 ZgsGevCm8NIx5BFRgCt+exllDFQx9hv6D6346NPPotLl/z5WP6NY6GZWguKzG2pO IWztPlSubZJS5dpqJqwCnMn3jDjFB1XgCCcPd6hRYPFYe5lP1qJm6qPuH+R9zpWD uxZ5iNkUf+tt0JFsb/1/1DthVWU1shGWVRU+P2XijUlg5eZQFtAc5x7eKu8UFDxh gt+zthA3MHkeadiqHyiIWTHDmDzM9Us7B3U9dBzdSp9g5Jmo9qPh+FqrU6oH9cdI Szhz+l8OTlLGPVh9an8XbNQMUSy1UDvBZB4GZ9qQbosHyPI5zZNbky0csuGdqMU= =jejh -----END PGP SIGNATURE-----

On 25 May 2016 at 14:52, David Caro <dcaro(a)redhat.com&gt; wrote: > On 05/25 14:42, Barak Korren wrote: >> On 25 May 2016 at 12:44, Eyal Edri <eedri(a)redhat.com&gt; wrote: >> > OK, >> > I suggest to test using a VM with local disk (preferably on a host w=

>> > configured), if its working, >> > lets expedite moving all VMs or at least a large amount of VMs to it=

>> > we see network load reduced. >> > >> >> This is not that easy, oVirt doesn't support mixing local disk and >> storage in the same cluster, so we will need to move hosts to a new >> cluster for this. >> Also we will lose the ability to use templates, or otherwise have to >> create the templates on each and every disk. >> >> The scratch disk is a good solution for this, where you can have the >> OS image on the central storage and the ephemeral data on the local >> disk. >> >> WRT to the storage architecture - a single huge (10.9T) ext4 is used >> as the FS on top of the DRBD, this is probably not the most efficient >> thing one can do (XFS would probably have been better, RAW via iSCSI - >> even better). > > That was done >3 years ago, xfs was not quite stable and widely used and > supported back then. > AFAIK it pre-dates EXT4

in any case this does not detract from the fact that the current configuration in not as efficient as we can make it. =20

=20 >> >> I'm guessing that those 10/9TB are not made from a single disk but >> with a hardware RAID of some sort. In this case deactivating the >> hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That >> are then re-joined to a single sotrage domain in oVirt) will enable >> different VMs to concurrently work on different disks. This should >> lower the per-vm storage latency. > > That would get rid of the drbd too, it's a totally different setup, from > scratch (no nfs either). =20 We can and should still use DRBD, just setup a device for each disk. But yeah, NFS should probably go away. (We are seeing dramatically better performance for iSCSI in integration-engine)

=20 > >> >> Looking at the storage machine I see strong indication it is IO bound >> - the load average is ~12 while there are just 1-5 working processes >> and the CPU is ~80% idle and the rest is IO wait. >> >> Running 'du *' at: >> /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/ima=

>> one can see that most images are ~40G in size (that is _real_ 40G not >> sparse!). This means that despite having most VMs created based on >> templates, the VMs are full template copies rather then COW clones. > > That should not be like that, maybe the templates are wrongly configure=

> foreman images? =20 This is the expected behaviour when creating a VM from template in the oVirt admin UI. I thought Foreman might behave differently, but it seems it does not. =20 This behaviour is determined by the parameters you pass to the engine API when instantiating a VM, so it most probably doesn't have anything to do with the template configuration.

=20 > >> What this means is that using pools (where all VMs are COW copies of >> the single pool template) is expected to significantly reduce the >> storage utilization and therefore the IO load on it (the less you >> store, the less you need to read back). > > That should happen too without pools, with normal qcow templates. =20 Not unless you create all the VMs via the API and pass the right parameters. Pools are the easiest way to ensure you never mess that up...

=20 > And in any case, that will not lower the normal io, when not actually > creating vms, as any read and write will still hit the disk anyhow, it > only alleviates the io when creating new vms. =20 Since you are reading the same bits over and over (for different VMs) you enable the various buffer caches along the way (in the storage machines and in the hypevirsors) to do what they are supposed to.

=20 > The local disk (scratch disk) is the best option > imo, now and for the foreseeable future. =20 This is not an either/or thing, IMO we need to do both.

=20 --=20 Barak Korren bkorren(a)redhat.com RHEV-CI Team

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRb9/AAoJEEBxx+HSYmnDNcMIAJ/VlG6dy+ggdTB0qQ8qyP/B lCJwJPEhG6vw6lwyGO0jxFAAMafcxaFvotAYpHX3A39EMAaoLpsvFYUhF8oJrAD6 9q3jDyuqbxkJ4/HLW2B4MFaB24911KcrYoreNMvgjT+iqE70flT45bfgG5OvgXdN 6LkrQ7x+iz+6aFYsTX/Vry2W84FVHZiACltckqkqjitkPBJgSTWv2zGXxF4Vd717 YrW404GoEU0v6Qy8V1s3y8BViH+8U0rkfHWRiSWohPxoAiLQfqlzQ+3SRtvipf9K vlxRSwoyZYM2iFwD1OnN9ZdU+2z2LFaew5HoK52/srefkgk3O0jIykJ3wdNdz+c= =nsl3 -----END PGP SIGNATURE-----

On 05/25 17:06, David Caro wrote: > On 05/25 16:09, Barak Korren wrote: > > On 25 May 2016 at 14:52, David Caro <dcaro(a)redhat.com&gt; wrote: > > > On 05/25 14:42, Barak Korren wrote: > > >> On 25 May 2016 at 12:44, Eyal Edri <eedri(a)redhat.com&gt; wrote: > > >> > OK, > > >> > I suggest to test using a VM with local disk (preferably on a host with SSD > > >> > configured), if its working, > > >> > lets expedite moving all VMs or at least a large amount of VMs to it until > > >> > we see network load reduced. > > >> > > > >> > > >> This is not that easy, oVirt doesn't support mixing local disk and > > >> storage in the same cluster, so we will need to move hosts to a new > > >> cluster for this. > > >> Also we will lose the ability to use templates, or otherwise have to > > >> create the templates on each and every disk. > > >> > > >> The scratch disk is a good solution for this, where you can have the > > >> OS image on the central storage and the ephemeral data on the local > > >> disk. > > >> > > >> WRT to the storage architecture - a single huge (10.9T) ext4 is used > > >> as the FS on top of the DRBD, this is probably not the most efficient > > >> thing one can do (XFS would probably have been better, RAW via iSCSI - > > >> even better). > > > > > > That was done >3 years ago, xfs was not quite stable and widely used and > > > supported back then. > > > > > AFAIK it pre-dates EXT4 > > It does, but for el6, it was performing way poorly, and with more bugs (for > what the reviews of it said at the time). > > > in any case this does not detract from the > > fact that the current configuration in not as efficient as we can make > > it. > > > > It does not, I agree to better focus on what we can do now on, now what should > have been done then. > > > > > >> > > >> I'm guessing that those 10/9TB are not made from a single disk but > > >> with a hardware RAID of some sort. In this case deactivating the > > >> hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That > > >> are then re-joined to a single sotrage domain in oVirt) will enable > > >> different VMs to concurrently work on different disks. This should > > >> lower the per-vm storage latency. > > > > > > That would get rid of the drbd too, it's a totally different setup, from > > > scratch (no nfs either). > > > > We can and should still use DRBD, just setup a device for each disk. > > But yeah, NFS should probably go away. > > (We are seeing dramatically better performance for iSCSI in > > integration-engine) > > I don't understand then what you said about splitting the hardware raids, you > mean to setup one drdb device on top of each hard drive instead? Though I really think we should move to gluster/ceph instead for the jenkins vms, anyone knows what's the current status of the hyperconverge?

That would allow us for better scalable distributed storage, and properly use the hosts local disks (we have more space on the combined hosts right now that on the storage servers).

> > > btw. I think that the nfs is used also for something more than just the engine > storage domain (just to keep it in mind that it has to be checked if we are > going to get rid of it) > > > > > > > > >> > > >> Looking at the storage machine I see strong indication it is IO bound > > >> - the load average is ~12 while there are just 1-5 working processes > > >> and the CPU is ~80% idle and the rest is IO wait. > > >> > > >> Running 'du *' at: > > >> /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images > > >> one can see that most images are ~40G in size (that is _real_ 40G not > > >> sparse!). This means that despite having most VMs created based on > > >> templates, the VMs are full template copies rather then COW clones. > > > > > > That should not be like that, maybe the templates are wrongly configured? or > > > foreman images? > > > > This is the expected behaviour when creating a VM from template in the > > oVirt admin UI. I thought Foreman might behave differently, but it > > seems it does not. > > > > This behaviour is determined by the parameters you pass to the engine > > API when instantiating a VM, so it most probably doesn't have anything > > to do with the template configuration. > > So maybe a misconfiguration in foreman? > > > > > > > > >> What this means is that using pools (where all VMs are COW copies of > > >> the single pool template) is expected to significantly reduce the > > >> storage utilization and therefore the IO load on it (the less you > > >> store, the less you need to read back). > > > > > > That should happen too without pools, with normal qcow templates. > > > > Not unless you create all the VMs via the API and pass the right > > parameters. Pools are the easiest way to ensure you never mess that > > up... > > That was the idea > > > > > > And in any case, that will not lower the normal io, when not actually > > > creating vms, as any read and write will still hit the disk anyhow, it > > > only alleviates the io when creating new vms. > > > > Since you are reading the same bits over and over (for different VMs) > > you enable the various buffer caches along the way (in the storage > > machines and in the hypevirsors) to do what they are supposed to. > > > Once the vm is started, mostly all that's needed is on ram, so there are not > that much reads from disk, unless you start writing down to it, and that's > mostly what we are hitting, lots of writes. > > > > > > The local disk (scratch disk) is the best option > > > imo, now and for the foreseeable future. > > > > This is not an either/or thing, IMO we need to do both. > > I think that it's way more useful, because it will solve our current issues > faster and for longer, so IMO it should get more attention sooner. > > Any improvement that does not remove the current bottleneck, is not really > giving any value to the overall infra (even if it might become valuable later). > > > > > -- > > Barak Korren > > bkorren(a)redhat.com > > RHEV-CI Team > > -- > David Caro > > Red Hat S.L. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Tel.: +420 532 294 605 > Email: dcaro(a)redhat.com > IRC: dcaro|dcaroest@{freenode|oftc|redhat} > Web: www.redhat.com > RHT Global #: 82-62605 -- David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro(a)redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605

> > > I agree a stable distributed storage solution is the way to go if we can > find one :) > =20 Distributed storages usually suffer from a large overhead because: 1. They try to be resilient to node failure, which means keeping two or more copies of the same file, which results in I/O overhead. 2. They need to coordinate metadata access for large amounts of files. Bottlenecks in the metadata management system are a common issue for distributes FS storages. =20 Since most of our data is ephemeral anyway I don't think we need to pay this overhead.

=20 =20 --=20 Barak Korren bkorren(a)redhat.com RHEV-CI Team

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRqqdAAoJEEBxx+HSYmnDFi8IAIwiMdFKkr9PxGf9T6uYlaG7 mOgJOR2cJmOSLZ1Ysl1HZp0GxHtFqplaIv8ZUOCQ2vweRCCg75EIyCOFSBJD8aUh /ZoYxtGkQA8r3KER+XDD9uoISvjKa9Ilh/OsgQ+fvP5VpB2tI59YRCFTBS6x5mte lfWWfb5HCoVpbY7rhI+sYK4xT7YyE68PnutCOkJ/HmAFJ8p3gtW43ci70KHb5ohF 6181qkDor9ZObdulIHkIQ8lDDMFUqqnjFpIlQEXhT5qSW93wZ3dx57sm9Bi9oyuu fwmupauMWp1R8A+v/NRD/33LHd1pD4WHifOFHV0Fp0XSniTTkhFO+wcWxwypF90= =mqNm -----END PGP SIGNATURE-----