ngn build jobs take more than twice (x) as long as in the last days
Hey, $subj says it all. Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ I.e. 3.6 - before: ~46min, now 1:23hrs In master it's even worse: >1:30hrs Can someone help to idnetify the reason? - fabian -- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat
--q8dntDJTu318bll0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/24 17:57, Fabian Deutsch wrote:
Hey, =20 $subj says it all. =20 Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ =20 I.e. 3.6 - before: ~46min, now 1:23hrs =20 In master it's even worse: >1:30hrs =20 Can someone help to idnetify the reason?
I see that this is where there's a big jump in time: 06:39:38 Domain installation still in progress. You can reconnect to=20 06:39:38 the console to complete the installation process. 07:21:51 ..................................................................= =2E........................................................................= =2E........................................................................= =2E........................................2016-05-24 03:21:51,341: Install= finished. Or at least virt shut down. So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to wor= k.
=20 - fabian =20 --=20 Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --q8dntDJTu318bll0 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRHs+AAoJEEBxx+HSYmnDrOMH/2LWGXriaZ2TpyotcgLYfbAI QKmlZdcbH1i0MgQDygjYHLw8qbo+1baTgW0thIt8LeCBKDdGahIAYUxVGYcPbJ17 jzOZpRNDs2gmBPeU8L3YRc2of5nQlD4ViunaoXjFSVDnhx2xIfIJPJhmi1/nquFn lyPxcTFMqC9dZz9dmEQ36Ua/v6GV35eY21dxNzbQh6kaXFwxsTCp6+eBosvFIn8Z UK9y2IPmC0Z9GMXxQnHxy/m2eaBj9Ad/nbqdgRrxgk42XHPm7N0otujpKkwJ0pui XLS3p6JPuuLf5UwPpE4pOxQ8Jo0lXZ1aTqX/MZmF4tjcMiMTgENHh1pKiuisVkg= =RMMp -----END PGP SIGNATURE----- --q8dntDJTu318bll0--
--i6vqABX3nJKXLk01 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/24 18:03, David Caro wrote:
On 05/24 17:57, Fabian Deutsch wrote:
Hey, =20 $subj says it all. =20 Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ =20 I.e. 3.6 - before: ~46min, now 1:23hrs =20 In master it's even worse: >1:30hrs =20 Can someone help to idnetify the reason? =20 =20 I see that this is where there's a big jump in time: =20 06:39:38 Domain installation still in progress. You can reconnect to=20 06:39:38 the console to complete the installation process. 07:21:51 ................................................................= =2E........................................................................= =2E........................................................................= =2E..........................................2016-05-24 03:21:51,341: Insta= ll finished. Or at least virt shut down. =20 So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to w= ork.
It seems that the virt-install log is not being archived, and the workdir h= as already been cleaned up so I can check the logfile: /home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86= _64/ovirt-node-ng/virt-install.log Maybe you can archive it too on the next run to debug
=20
=20 - fabian =20 --=20 Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --i6vqABX3nJKXLk01 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRHwPAAoJEEBxx+HSYmnD568IAIgydp+kOT7lxkmqGxT8fR1Q X4I7RBHDCrZwnXmtaeebxtV3U7EwkorIEQiKXJVgqwJM5qtqeLTQUXNd8MU/sco5 Hz4jLuydM1W7amVPfj6ka4ZBemy6pghNF+3zRdUgMFQwFwMsQ9/anAZK/sAcO0/M +NToKWNbG/cCVj3Dx24ATDUYvXjDxn27yp9b7e0etswliO+/0Jb9XZOwnZwU8SKu hXIy1x5e/3c8pwMrli/TI5vPxns5A9ubEEFmlWzczhHz7y/RrdcxkNmfNHLGeawx nxpq1Kidxh8NUqxoZ41P/HVO6Bm7Ir5T2JOs+XhKii1eLvH13m6EMWCmpeNnSwY= =OtiM -----END PGP SIGNATURE----- --i6vqABX3nJKXLk01--
Is it running on the same slave as before? Can we see a changelog of things changed from the Last time it took less time? On May 24, 2016 7:06 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/24 18:03, David Caro wrote:
On 05/24 17:57, Fabian Deutsch wrote:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I see that this is where there's a big jump in time:
06:39:38 Domain installation still in progress. You can reconnect to 06:39:38 the console to complete the installation process. 07:21:51 .............................................................................................................................................................................................................................................................2016-05-24 03:21:51,341: Install finished. Or at least virt shut down.
So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to work.
It seems that the virt-install log is not being archived, and the workdir has already been cleaned up so I can check the logfile:
/home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log
Maybe you can archive it too on the next run to debug
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
I don't know if it was quicker on different salves, because I can not look at the history. There were no Node sided changes, maybe the package set in master got larger - well see that on the first regular build. But: I enabled kvm in mock, and this seems to help. The first job post this chnage looks good. My assumption is that previous builds were running on a beefier slave. - fabian On Tue, May 24, 2016 at 6:50 PM, Eyal Edri <eedri@redhat.com> wrote:
Is it running on the same slave as before? Can we see a changelog of things changed from the Last time it took less time?
On May 24, 2016 7:06 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/24 18:03, David Caro wrote:
On 05/24 17:57, Fabian Deutsch wrote:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I see that this is where there's a big jump in time:
06:39:38 Domain installation still in progress. You can reconnect to 06:39:38 the console to complete the installation process. 07:21:51 .............................................................................................................................................................................................................................................................2016-05-24 03:21:51,341: Install finished. Or at least virt shut down.
So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to work.
It seems that the virt-install log is not being archived, and the workdir has already been cleaned up so I can check the logfile:
/home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log
Maybe you can archive it too on the next run to debug
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat
btw, feel free to increase history for node builds now, we have the space for it. I wouldn't got crazy with keeping history of artifacts, but build info with console output there is no problem keeping a month old. I'm adding Barak as this weekly infra owner to help debug this tomorrow if needed. Also, Evgheni is working on network configuration, might be relevant as well. On Tue, May 24, 2016 at 9:30 PM, Fabian Deutsch <fdeutsch@redhat.com> wrote:
I don't know if it was quicker on different salves, because I can not look at the history.
There were no Node sided changes, maybe the package set in master got larger - well see that on the first regular build.
But: I enabled kvm in mock, and this seems to help. The first job post this chnage looks good.
My assumption is that previous builds were running on a beefier slave.
- fabian
Is it running on the same slave as before? Can we see a changelog of
On Tue, May 24, 2016 at 6:50 PM, Eyal Edri <eedri@redhat.com> wrote: things
changed from the Last time it took less time?
On May 24, 2016 7:06 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/24 18:03, David Caro wrote:
On 05/24 17:57, Fabian Deutsch wrote:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I see that this is where there's a big jump in time:
06:39:38 Domain installation still in progress. You can reconnect to 06:39:38 the console to complete the installation process. 07:21:51
.............................................................................................................................................................................................................................................................2016-05-24
03:21:51,341: Install finished. Or at least virt shut down.
So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to work.
It seems that the virt-install log is not being archived, and the workdir has already been cleaned up so I can check the logfile:
/home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log
Maybe you can archive it too on the next run to debug
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Tue, May 24, 2016 at 8:49 PM, Eyal Edri <eedri@redhat.com> wrote:
btw, feel free to increase history for node builds now, we have the space for it. I wouldn't got crazy with keeping history of artifacts, but build info with console output there is no problem keeping a month old.
~2GB * 3 * 30 == 180GB of disk space, ok?
I'm adding Barak as this weekly infra owner to help debug this tomorrow if needed. Also, Evgheni is working on network configuration, might be relevant as well.
Actually, one thought: I think it might make sense to change the standard job template to support nesting. Currently i need to create /dev/kvm to get at least that acceleration. (If the app inside mock s speaking to /dev/kvm directly). Also for libvirt a mount is needed. My take would be: Change the standard ci job template so that every job can use libvirt + geustfosh (w/ acceleration) right away without needing these hacks. If that sounds viable, then I can open a ticket. - fabain
On Tue, May 24, 2016 at 9:30 PM, Fabian Deutsch <fdeutsch@redhat.com> wrote:
I don't know if it was quicker on different salves, because I can not look at the history.
There were no Node sided changes, maybe the package set in master got larger - well see that on the first regular build.
But: I enabled kvm in mock, and this seems to help. The first job post this chnage looks good.
My assumption is that previous builds were running on a beefier slave.
- fabian
On Tue, May 24, 2016 at 6:50 PM, Eyal Edri <eedri@redhat.com> wrote:
Is it running on the same slave as before? Can we see a changelog of things changed from the Last time it took less time?
On May 24, 2016 7:06 PM, "David Caro" <dcaro@redhat.com> wrote:
On 05/24 18:03, David Caro wrote:
On 05/24 17:57, Fabian Deutsch wrote:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I see that this is where there's a big jump in time:
06:39:38 Domain installation still in progress. You can reconnect to 06:39:38 the console to complete the installation process. 07:21:51
.............................................................................................................................................................................................................................................................2016-05-24 03:21:51,341: Install finished. Or at least virt shut down.
So it looks as if the code that checks if the domain is shut down is not working properly, or maybe the virt-install is taking very very long to work.
It seems that the virt-install log is not being archived, and the workdir has already been cleaned up so I can check the logfile:
/home/jenkins/workspace/ovirt-node-ng_ovirt-3.6_build-artifacts-fc22-x86_64/ovirt-node-ng/virt-install.log
Maybe you can archive it too on the next run to debug
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat
Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch@redhat.com> ha scritto:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I have no numbers but I have the feeling that all jobs are getting slower since a couple of weeks ago. Yum install phase takes ages. I thoughtit was some temporary storage i/o peak but looks like it's not temporary.
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
It might be more load on the storage servers with now running much more jobs. Evgheni - can you check if the load on the storage servers has changed significantly to justify this degradation of service? We need to expedite the enablement of SSDs in the hypervisors and move to local hooks. Anton - do we have a test VM that uses a local DISK we can use to test if it improves the runtime? On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch@redhat.com> ha scritto:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I have no numbers but I have the feeling that all jobs are getting slower since a couple of weeks ago. Yum install phase takes ages. I thoughtit was some temporary storage i/o peak but looks like it's not temporary.
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Fabian, Please open a ticket to enable all VMs to support nested, I don't see a reason why not to. Evgheni - we'll need to make sure all hosts supports nested, it will require reboot of a server if its not enabled now. On Wed, May 25, 2016 at 10:31 AM, Eyal Edri <eedri@redhat.com> wrote:
It might be more load on the storage servers with now running much more jobs. Evgheni - can you check if the load on the storage servers has changed significantly to justify this degradation of service?
We need to expedite the enablement of SSDs in the hypervisors and move to local hooks. Anton - do we have a test VM that uses a local DISK we can use to test if it improves the runtime?
On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch@redhat.com> ha scritto:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I have no numbers but I have the feeling that all jobs are getting slower since a couple of weeks ago. Yum install phase takes ages. I thoughtit was some temporary storage i/o peak but looks like it's not temporary.
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On 25 May 2016 at 10:33, Eyal Edri <eedri@redhat.com> wrote:
Fabian, Please open a ticket to enable all VMs to support nested, I don't see a reason why not to. Evgheni - we'll need to make sure all hosts supports nested, it will require reboot of a server if its not enabled now.
The point is not to make VMs support nesting (they all already do probably) Its to configure std-ci to pre-configure KVM access for mock (without having the devs specify that in the automation dir) I'm not sure I'm in favour of that.
If that is the case, and the required need can be set in the automation dir the I agree. Is that possible to set via scripts in the automation dir? On Wed, May 25, 2016 at 10:59 AM, Barak Korren <bkorren@redhat.com> wrote:
On 25 May 2016 at 10:33, Eyal Edri <eedri@redhat.com> wrote:
Fabian, Please open a ticket to enable all VMs to support nested, I don't see a reason why not to. Evgheni - we'll need to make sure all hosts supports nested, it will require reboot of a server if its not enabled now.
The point is not to make VMs support nesting (they all already do probably) Its to configure std-ci to pre-configure KVM access for mock (without having the devs specify that in the automation dir) I'm not sure I'm in favour of that.
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On 25 May 2016 at 11:02, Eyal Edri <eedri@redhat.com> wrote:
If that is the case, and the required need can be set in the automation dir the I agree. Is that possible to set via scripts in the automation dir?
Yes. that is how Lago does it. -- Barak Korren bkorren@redhat.com RHEV-CI Team
Hi, I checked SAR data on storage servers and compared loads yesterday and three weeks ago (May 3). Load values are in pretty much the same range, yet now most of the time they are around the "high" mark so we may be nearing a bottleneck, specifically on I/O where we mostly do writes to the NAS, not reads and there's quite a bit of overhead: VM -> QCOW -> file -> network -> NFS -> DRBD -> disk Surely, using local scratch disks stored on SSDs should greatly improve performance as at least half of the above steps will be gone. We don't really need to centrally store (NFS) or mirror (DRBD) data that slaves write to their disks all the time anyways. For VMs where we do need redundancy, I'd suggest using iSCSI storage domains in the long run. Regards, Evgheni Dereveanchin ----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Sandro Bonazzola" <sbonazzo@redhat.com>, "Evgheni Dereveanchin" <ederevea@redhat.com>, "Anton Marchukov" <amarchuk@redhat.com> Cc: "Fabian Deutsch" <fdeutsch@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, 25 May, 2016 9:31:43 AM Subject: Re: ngn build jobs take more than twice (x) as long as in the last days It might be more load on the storage servers with now running much more jobs. Evgheni - can you check if the load on the storage servers has changed significantly to justify this degradation of service? We need to expedite the enablement of SSDs in the hypervisors and move to local hooks. Anton - do we have a test VM that uses a local DISK we can use to test if it improves the runtime? On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch@redhat.com> ha scritto:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I have no numbers but I have the feeling that all jobs are getting slower since a couple of weeks ago. Yum install phase takes ages. I thoughtit was some temporary storage i/o peak but looks like it's not temporary.
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
OK, I suggest to test using a VM with local disk (preferably on a host with SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it until we see network load reduced. e. On Wed, May 25, 2016 at 12:38 PM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi,
I checked SAR data on storage servers and compared loads yesterday and three weeks ago (May 3). Load values are in pretty much the same range, yet now most of the time they are around the "high" mark so we may be nearing a bottleneck, specifically on I/O where we mostly do writes to the NAS, not reads and there's quite a bit of overhead:
VM -> QCOW -> file -> network -> NFS -> DRBD -> disk
Surely, using local scratch disks stored on SSDs should greatly improve performance as at least half of the above steps will be gone. We don't really need to centrally store (NFS) or mirror (DRBD) data that slaves write to their disks all the time anyways.
For VMs where we do need redundancy, I'd suggest using iSCSI storage domains in the long run.
Regards, Evgheni Dereveanchin
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Sandro Bonazzola" <sbonazzo@redhat.com>, "Evgheni Dereveanchin" < ederevea@redhat.com>, "Anton Marchukov" <amarchuk@redhat.com> Cc: "Fabian Deutsch" <fdeutsch@redhat.com>, "infra" <infra@ovirt.org> Sent: Wednesday, 25 May, 2016 9:31:43 AM Subject: Re: ngn build jobs take more than twice (x) as long as in the last days
It might be more load on the storage servers with now running much more jobs. Evgheni - can you check if the load on the storage servers has changed significantly to justify this degradation of service?
We need to expedite the enablement of SSDs in the hypervisors and move to local hooks. Anton - do we have a test VM that uses a local DISK we can use to test if it improves the runtime?
On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 24/Mag/2016 17:57, "Fabian Deutsch" <fdeutsch@redhat.com> ha scritto:
Hey,
$subj says it all.
Affected jobs are: http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/
I.e. 3.6 - before: ~46min, now 1:23hrs
In master it's even worse: >1:30hrs
Can someone help to idnetify the reason?
I have no numbers but I have the feeling that all jobs are getting slower since a couple of weeks ago. Yum install phase takes ages. I thoughtit
was
some temporary storage i/o peak but looks like it's not temporary.
- fabian
-- Fabian Deutsch <fdeutsch@redhat.com> RHEV Hypervisor Red Hat _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On 25 May 2016 at 12:44, Eyal Edri <eedri@redhat.com> wrote:
OK, I suggest to test using a VM with local disk (preferably on a host with SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it until we see network load reduced.
This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk. The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk. WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI - even better). I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency. Looking at the storage machine I see strong indication it is IO bound - the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait. Running 'du *' at: /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones. What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back). -- Barak Korren bkorren@redhat.com RHEV-CI Team
--jigfid2yHjNFZUTO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/25 14:42, Barak Korren wrote:
On 25 May 2016 at 12:44, Eyal Edri <eedri@redhat.com> wrote:
OK, I suggest to test using a VM with local disk (preferably on a host with= SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it un= til we see network load reduced.
=20 This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk. =20 The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk. =20 WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI - even better).
That was done >3 years ago, xfs was not quite stable and widely used and supported back then.
=20 I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency.
That would get rid of the drbd too, it's a totally different setup, from scratch (no nfs either).
=20 Looking at the storage machine I see strong indication it is IO bound - the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait. =20 Running 'du *' at: /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones.
That should not be like that, maybe the templates are wrongly configured? or foreman images?
What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back).
That should happen too without pools, with normal qcow templates. And in any case, that will not lower the normal io, when not actually creat= ing vms, as any read and write will still hit the disk anyhow, it only alleviat= es the io when creating new vms. The local disk (scratch disk) is the best opt= ion imo, now and for the foreseeable future.
=20 --=20 Barak Korren bkorren@redhat.com RHEV-CI Team _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --jigfid2yHjNFZUTO Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRZIOAAoJEEBxx+HSYmnDtxoH/Rr9aRoqydoTprHsg7VEwNx0 ZgsGevCm8NIx5BFRgCt+exllDFQx9hv6D6346NPPotLl/z5WP6NY6GZWguKzG2pO IWztPlSubZJS5dpqJqwCnMn3jDjFB1XgCCcPd6hRYPFYe5lP1qJm6qPuH+R9zpWD uxZ5iNkUf+tt0JFsb/1/1DthVWU1shGWVRU+P2XijUlg5eZQFtAc5x7eKu8UFDxh gt+zthA3MHkeadiqHyiIWTHDmDzM9Us7B3U9dBzdSp9g5Jmo9qPh+FqrU6oH9cdI Szhz+l8OTlLGPVh9an8XbNQMUSy1UDvBZB4GZ9qQbosHyPI5zZNbky0csuGdqMU= =jejh -----END PGP SIGNATURE----- --jigfid2yHjNFZUTO--
On 25 May 2016 at 14:52, David Caro <dcaro@redhat.com> wrote:
On 05/25 14:42, Barak Korren wrote:
On 25 May 2016 at 12:44, Eyal Edri <eedri@redhat.com> wrote:
OK, I suggest to test using a VM with local disk (preferably on a host with SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it until we see network load reduced.
This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk.
The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk.
WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI - even better).
That was done >3 years ago, xfs was not quite stable and widely used and supported back then.
AFAIK it pre-dates EXT4, in any case this does not detract from the fact that the current configuration in not as efficient as we can make it.
I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency.
That would get rid of the drbd too, it's a totally different setup, from scratch (no nfs either).
We can and should still use DRBD, just setup a device for each disk. But yeah, NFS should probably go away. (We are seeing dramatically better performance for iSCSI in integration-engine)
Looking at the storage machine I see strong indication it is IO bound - the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait.
Running 'du *' at: /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones.
That should not be like that, maybe the templates are wrongly configured? or foreman images?
This is the expected behaviour when creating a VM from template in the oVirt admin UI. I thought Foreman might behave differently, but it seems it does not. This behaviour is determined by the parameters you pass to the engine API when instantiating a VM, so it most probably doesn't have anything to do with the template configuration.
What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back).
That should happen too without pools, with normal qcow templates.
Not unless you create all the VMs via the API and pass the right parameters. Pools are the easiest way to ensure you never mess that up...
And in any case, that will not lower the normal io, when not actually creating vms, as any read and write will still hit the disk anyhow, it only alleviates the io when creating new vms.
Since you are reading the same bits over and over (for different VMs) you enable the various buffer caches along the way (in the storage machines and in the hypevirsors) to do what they are supposed to.
The local disk (scratch disk) is the best option imo, now and for the foreseeable future.
This is not an either/or thing, IMO we need to do both. -- Barak Korren bkorren@redhat.com RHEV-CI Team
--qjXtncIm5b3tWrFJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/25 16:09, Barak Korren wrote:
On 25 May 2016 at 14:52, David Caro <dcaro@redhat.com> wrote:
On 05/25 14:42, Barak Korren wrote:
On 25 May 2016 at 12:44, Eyal Edri <eedri@redhat.com> wrote:
OK, I suggest to test using a VM with local disk (preferably on a host w= ith SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it= until we see network load reduced.
This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk.
The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk.
WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI - even better).
That was done >3 years ago, xfs was not quite stable and widely used and supported back then.
AFAIK it pre-dates EXT4
It does, but for el6, it was performing way poorly, and with more bugs (for what the reviews of it said at the time).
in any case this does not detract from the fact that the current configuration in not as efficient as we can make it. =20
It does not, I agree to better focus on what we can do now on, now what sho= uld have been done then.
=20
I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs (That are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency.
That would get rid of the drbd too, it's a totally different setup, from scratch (no nfs either). =20 We can and should still use DRBD, just setup a device for each disk. But yeah, NFS should probably go away. (We are seeing dramatically better performance for iSCSI in integration-engine)
I don't understand then what you said about splitting the hardware raids, y= ou mean to setup one drdb device on top of each hard drive instead?=20 btw. I think that the nfs is used also for something more than just the eng= ine storage domain (just to keep it in mind that it has to be checked if we are going to get rid of it)
=20
Looking at the storage machine I see strong indication it is IO bound - the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait.
Running 'du *' at: /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/ima=
ges
one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones.
That should not be like that, maybe the templates are wrongly configure= d? or foreman images? =20 This is the expected behaviour when creating a VM from template in the oVirt admin UI. I thought Foreman might behave differently, but it seems it does not. =20 This behaviour is determined by the parameters you pass to the engine API when instantiating a VM, so it most probably doesn't have anything to do with the template configuration.
So maybe a misconfiguration in foreman?
=20
What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back).
That should happen too without pools, with normal qcow templates.
=20 Not unless you create all the VMs via the API and pass the right parameters. Pools are the easiest way to ensure you never mess that up...
That was the idea
=20
And in any case, that will not lower the normal io, when not actually creating vms, as any read and write will still hit the disk anyhow, it only alleviates the io when creating new vms. =20 Since you are reading the same bits over and over (for different VMs) you enable the various buffer caches along the way (in the storage machines and in the hypevirsors) to do what they are supposed to.
Once the vm is started, mostly all that's needed is on ram, so there are not that much reads from disk, unless you start writing down to it, and that's mostly what we are hitting, lots of writes.
=20
The local disk (scratch disk) is the best option imo, now and for the foreseeable future. =20 This is not an either/or thing, IMO we need to do both.
I think that it's way more useful, because it will solve our current issues faster and for longer, so IMO it should get more attention sooner. Any improvement that does not remove the current bottleneck, is not really giving any value to the overall infra (even if it might become valuable lat= er).
=20 --=20 Barak Korren bkorren@redhat.com RHEV-CI Team
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --qjXtncIm5b3tWrFJ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRb9/AAoJEEBxx+HSYmnDNcMIAJ/VlG6dy+ggdTB0qQ8qyP/B lCJwJPEhG6vw6lwyGO0jxFAAMafcxaFvotAYpHX3A39EMAaoLpsvFYUhF8oJrAD6 9q3jDyuqbxkJ4/HLW2B4MFaB24911KcrYoreNMvgjT+iqE70flT45bfgG5OvgXdN 6LkrQ7x+iz+6aFYsTX/Vry2W84FVHZiACltckqkqjitkPBJgSTWv2zGXxF4Vd717 YrW404GoEU0v6Qy8V1s3y8BViH+8U0rkfHWRiSWohPxoAiLQfqlzQ+3SRtvipf9K vlxRSwoyZYM2iFwD1OnN9ZdU+2z2LFaew5HoK52/srefkgk3O0jIykJ3wdNdz+c= =nsl3 -----END PGP SIGNATURE----- --qjXtncIm5b3tWrFJ--
On 05/25 16:09, Barak Korren wrote:
On 25 May 2016 at 14:52, David Caro <dcaro@redhat.com> wrote:
On 05/25 14:42, Barak Korren wrote:
On 25 May 2016 at 12:44, Eyal Edri <eedri@redhat.com> wrote:
OK, I suggest to test using a VM with local disk (preferably on a host= with SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to = it until we see network load reduced.
This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk.
The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk.
WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI=
--iiZKCn1f/U0ES2iY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/25 17:06, David Caro wrote: -
even better).
That was done >3 years ago, xfs was not quite stable and widely used = and supported back then.
AFAIK it pre-dates EXT4 =20 It does, but for el6, it was performing way poorly, and with more bugs (f= or what the reviews of it said at the time). =20 in any case this does not detract from the fact that the current configuration in not as efficient as we can make it. =20 =20 It does not, I agree to better focus on what we can do now on, now what s= hould have been done then. =20 =20
I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs (Th=
at
are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency.
That would get rid of the drbd too, it's a totally different setup, f= rom scratch (no nfs either). =20 We can and should still use DRBD, just setup a device for each disk. But yeah, NFS should probably go away. (We are seeing dramatically better performance for iSCSI in integration-engine) =20 I don't understand then what you said about splitting the hardware raids,= you mean to setup one drdb device on top of each hard drive instead?=20
Though I really think we should move to gluster/ceph instead for the jenkins vms, anyone knows what's the current status of the hyperconverge? That would allow us for better scalable distributed storage, and properly u= se the hosts local disks (we have more space on the combined hosts right now t= hat on the storage servers).
=20
Looking at the storage machine I see strong indication it is IO bound - the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait.
Running 'du *' at: /srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/i=
one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones.
That should not be like that, maybe the templates are wrongly configu= red? or foreman images? =20 This is the expected behaviour when creating a VM from template in the oVirt admin UI. I thought Foreman might behave differently, but it seems it does not. =20 This behaviour is determined by the parameters you pass to the engine API when instantiating a VM, so it most probably doesn't have anything to do with the template configuration. =20 So maybe a misconfiguration in foreman? =20 =20
What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back).
That should happen too without pools, with normal qcow templates. =20 Not unless you create all the VMs via the API and pass the right
mages parameters. Pools are the easiest way to ensure you never mess that up... =20 That was the idea =20 =20
And in any case, that will not lower the normal io, when not actually creating vms, as any read and write will still hit the disk anyhow, it only alleviates the io when creating new vms. =20 Since you are reading the same bits over and over (for different VMs) you enable the various buffer caches along the way (in the storage machines and in the hypevirsors) to do what they are supposed to. =20 =20 Once the vm is started, mostly all that's needed is on ram, so there are = not
=20 =20 btw. I think that the nfs is used also for something more than just the e= ngine storage domain (just to keep it in mind that it has to be checked if we a= re going to get rid of it) =20 that much reads from disk, unless you start writing down to it, and that's mostly what we are hitting, lots of writes. =20
=20
The local disk (scratch disk) is the best option imo, now and for the foreseeable future. =20 This is not an either/or thing, IMO we need to do both. =20 I think that it's way more useful, because it will solve our current issu= es faster and for longer, so IMO it should get more attention sooner. =20 Any improvement that does not remove the current bottleneck, is not really giving any value to the overall infra (even if it might become valuable l= ater). =20 =20 --=20 Barak Korren bkorren@redhat.com RHEV-CI Team =20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --iiZKCn1f/U0ES2iY Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRcH2AAoJEEBxx+HSYmnDh3UH/08y1/p2/Xf5jzMSRTbnSnZQ GTZNyseiTzG2G7gWj7wkMWkSiLbl3vCXOGppt4EhWkGBaBs6jEfaEqDB8yCRg88T dNew80UTRW73EQXR4HwriYzm0zSeGGU4Y68Bg98yBB4jeuaO4B5uwNzNMjdwMgXs Gg2j9rpEyI/hsS2qsVw9l2uA8Q7mQ6QZqL/m/zeEZL4xfHT65685vrPt4XmrLCpb 39RY7jtcLdF2rk9UhLPmaLN5wTGAjKooo/5KdcuQ/uM1x1oPLZtzQssMVX6ZD83A KqK/4aDrUivyz6IGg3kgOwfdcK38qAAvJmSxRCfaM2ynfVyeBmAY5VCp+JocEsg= =r1Oa -----END PGP SIGNATURE----- --iiZKCn1f/U0ES2iY--
On Wed, May 25, 2016 at 6:17 PM, David Caro <dcaro@redhat.com> wrote:
On 05/25 17:06, David Caro wrote:
On 05/25 16:09, Barak Korren wrote:
On 25 May 2016 at 14:52, David Caro <dcaro@redhat.com> wrote:
On 05/25 14:42, Barak Korren wrote:
On 25 May 2016 at 12:44, Eyal Edri <eedri@redhat.com> wrote:
OK, I suggest to test using a VM with local disk (preferably on a host with SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it until we see network load reduced.
This is not that easy, oVirt doesn't support mixing local disk and storage in the same cluster, so we will need to move hosts to a new cluster for this. Also we will lose the ability to use templates, or otherwise have to create the templates on each and every disk.
The scratch disk is a good solution for this, where you can have the OS image on the central storage and the ephemeral data on the local disk.
WRT to the storage architecture - a single huge (10.9T) ext4 is used as the FS on top of the DRBD, this is probably not the most efficient thing one can do (XFS would probably have been better, RAW via iSCSI - even better).
That was done >3 years ago, xfs was not quite stable and widely used and supported back then.
AFAIK it pre-dates EXT4
It does, but for el6, it was performing way poorly, and with more bugs (for what the reviews of it said at the time).
in any case this does not detract from the fact that the current configuration in not as efficient as we can make it.
It does not, I agree to better focus on what we can do now on, now what should have been done then.
I'm guessing that those 10/9TB are not made from a single disk but with a hardware RAID of some sort. In this case deactivating the hardware RAID and re-exposing it as multiple separate iSCSI LUNs
(That
are then re-joined to a single sotrage domain in oVirt) will enable different VMs to concurrently work on different disks. This should lower the per-vm storage latency.
That would get rid of the drbd too, it's a totally different setup, from scratch (no nfs either).
We can and should still use DRBD, just setup a device for each disk. But yeah, NFS should probably go away. (We are seeing dramatically better performance for iSCSI in integration-engine)
I don't understand then what you said about splitting the hardware raids, you mean to setup one drdb device on top of each hard drive instead?
Though I really think we should move to gluster/ceph instead for the jenkins vms, anyone knows what's the current status of the hyperconverge?
Neither Gluster nor Hyper-converge I think is stable enough to move all production infra into. Hyperconverged is not supported yet for oVirt as well (might be a 4.x feature)
That would allow us for better scalable distributed storage, and properly use the hosts local disks (we have more space on the combined hosts right now that on the storage servers).
I agree a stable distributed storage solution is the way to go if we can find one :)
btw. I think that the nfs is used also for something more than just the
storage domain (just to keep it in mind that it has to be checked if we are going to get rid of it)
Looking at the storage machine I see strong indication it is IO
bound
- the load average is ~12 while there are just 1-5 working processes and the CPU is ~80% idle and the rest is IO wait.
Running 'du *' at:
/srv/ovirt_storage/jenkins-dc/658e5b87-1207-4226-9fcc-4e5fa02b86b4/images
one can see that most images are ~40G in size (that is _real_ 40G not sparse!). This means that despite having most VMs created based on templates, the VMs are full template copies rather then COW clones.
That should not be like that, maybe the templates are wrongly configured? or foreman images?
This is the expected behaviour when creating a VM from template in the oVirt admin UI. I thought Foreman might behave differently, but it seems it does not.
This behaviour is determined by the parameters you pass to the engine API when instantiating a VM, so it most probably doesn't have anything to do with the template configuration.
So maybe a misconfiguration in foreman?
What this means is that using pools (where all VMs are COW copies of the single pool template) is expected to significantly reduce the storage utilization and therefore the IO load on it (the less you store, the less you need to read back).
That should happen too without pools, with normal qcow templates.
Not unless you create all the VMs via the API and pass the right parameters. Pools are the easiest way to ensure you never mess that up...
That was the idea
And in any case, that will not lower the normal io, when not actually creating vms, as any read and write will still hit the disk anyhow,
it
only alleviates the io when creating new vms.
Since you are reading the same bits over and over (for different VMs) you enable the various buffer caches along the way (in the storage machines and in the hypevirsors) to do what they are supposed to.
Once the vm is started, mostly all that's needed is on ram, so there are not that much reads from disk, unless you start writing down to it, and
engine that's
mostly what we are hitting, lots of writes.
The local disk (scratch disk) is the best option imo, now and for the foreseeable future.
This is not an either/or thing, IMO we need to do both.
I think that it's way more useful, because it will solve our current issues faster and for longer, so IMO it should get more attention sooner.
Any improvement that does not remove the current bottleneck, is not really giving any value to the overall infra (even if it might become valuable later).
-- Barak Korren bkorren@redhat.com RHEV-CI Team
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605
-- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
I agree a stable distributed storage solution is the way to go if we can find one :)
Distributed storages usually suffer from a large overhead because: 1. They try to be resilient to node failure, which means keeping two or more copies of the same file, which results in I/O overhead. 2. They need to coordinate metadata access for large amounts of files. Bottlenecks in the metadata management system are a common issue for distributes FS storages. Since most of our data is ephemeral anyway I don't think we need to pay this overhead. -- Barak Korren bkorren@redhat.com RHEV-CI Team
--+VeMz1SRxFChf6vr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 05/26 10:20, Barak Korren wrote:
I agree a stable distributed storage solution is the way to go if we can find one :)
=20 Distributed storages usually suffer from a large overhead because: 1. They try to be resilient to node failure, which means keeping two or more copies of the same file, which results in I/O overhead. 2. They need to coordinate metadata access for large amounts of files. Bottlenecks in the metadata management system are a common issue for distributes FS storages. =20 Since most of our data is ephemeral anyway I don't think we need to pay this overhead.
The solution for our current temporary ephemeral data would be for each node to create the vms locally, that's the scratch disks solution we started wit= h. The distributed storage would be used to store the jenkins machines templat= es, that mostly would be read by the hosts, and thus, properly cached locally w= ith a low miss rate (as they don't usually change). To actually not use at all = the central storage, whose extra levels of redundancy are only useful for more critical data (aka production datacenter machines).
=20 =20 --=20 Barak Korren bkorren@redhat.com RHEV-CI Team
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com IRC: dcaro|dcaroest@{freenode|oftc|redhat} Web: www.redhat.com RHT Global #: 82-62605 --+VeMz1SRxFChf6vr Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXRqqdAAoJEEBxx+HSYmnDFi8IAIwiMdFKkr9PxGf9T6uYlaG7 mOgJOR2cJmOSLZ1Ysl1HZp0GxHtFqplaIv8ZUOCQ2vweRCCg75EIyCOFSBJD8aUh /ZoYxtGkQA8r3KER+XDD9uoISvjKa9Ilh/OsgQ+fvP5VpB2tI59YRCFTBS6x5mte lfWWfb5HCoVpbY7rhI+sYK4xT7YyE68PnutCOkJ/HmAFJ8p3gtW43ci70KHb5ohF 6181qkDor9ZObdulIHkIQ8lDDMFUqqnjFpIlQEXhT5qSW93wZ3dx57sm9Bi9oyuu fwmupauMWp1R8A+v/NRD/33LHd1pD4WHifOFHV0Fp0XSniTTkhFO+wcWxwypF90= =mqNm -----END PGP SIGNATURE----- --+VeMz1SRxFChf6vr--
participants (6)
-
Barak Korren -
David Caro -
Evgheni Dereveanchin -
Eyal Edri -
Fabian Deutsch -
Sandro Bonazzola