[VDSM] All tests using directio fail on CI
Hi all, It seems that the CI setup has changed, and /var/tmp is using now tempfs. This is not compatible with vdsm tests, assuming that /var/tmp is a real file system. This is the reason we do not use /tmp. We have lot of storage tests using directio, and directio cannot work on tempfs. Please check the slaves and make sure /var/tmp is using file system supporting directio. See example failure bellow. Nir ---- 12:33:20 ====================================================================== 12:33:20 ERROR: test_create_fail_creating_lease (storage_volume_artifacts_test.BlockVolumeArtifactsTests) 12:33:20 ---------------------------------------------------------------------- 12:33:20 Traceback (most recent call last): 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", line 485, in test_create_fail_creating_lease 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in assertRaises 12:33:20 callableObj(*args, **kwargs) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 391, in create 12:33:20 desc, parent) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 482, in _create_metadata 12:33:20 sc.LEGAL_VOL) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 427, in newMetadata 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 420, in createMetadata 12:33:20 cls._putMetadata(metaId, meta) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", line 242, in _putMetadata 12:33:20 f.write(data) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", line 161, in write 12:33:20 raise OSError(err, msg) 12:33:20 OSError: [Errno 22] Invalid argument 12:33:20 -------------------- >> begin captured logging << -------------------- 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] (MainThread) Created a persistent dict with VGTagMetadataRW backend 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) read lines (VGTagMetadataRW)=[] 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Empty metadata 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Starting transaction 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) Flushing changes 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, MDT_VERSION=3 removing= 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] (MainThread) Finished transaction 12:33:20 2016-09-28 05:32:03,261 INFO [storage.BlockVolumeArtifacts] (MainThread) Create placeholder /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 for image's volumes 12:33:20 2016-09-28 05:32:03,264 WARNING [storage.StorageDomainManifest] (MainThread) Could not find mapping for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 12:33:20 2016-09-28 05:32:03,264 DEBUG [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG c088020e-91b2-45e6-85fd-eea3fce58764 12:33:20 --------------------- >> end captured logging << --------------------- 1
The CI setup did not change recently. All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go. But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail? On 28 September 2016 at 22:00, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
It seems that the CI setup has changed, and /var/tmp is using now tempfs.
This is not compatible with vdsm tests, assuming that /var/tmp is a real file system. This is the reason we do not use /tmp.
We have lot of storage tests using directio, and directio cannot work on tempfs.
Please check the slaves and make sure /var/tmp is using file system supporting directio.
See example failure bellow.
Nir
----
12:33:20 ====================================================================== 12:33:20 ERROR: test_create_fail_creating_lease (storage_volume_artifacts_test.BlockVolumeArtifactsTests) 12:33:20 ---------------------------------------------------------------------- 12:33:20 Traceback (most recent call last): 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", line 485, in test_create_fail_creating_lease 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in assertRaises 12:33:20 callableObj(*args, **kwargs) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 391, in create 12:33:20 desc, parent) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 482, in _create_metadata 12:33:20 sc.LEGAL_VOL) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 427, in newMetadata 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 420, in createMetadata 12:33:20 cls._putMetadata(metaId, meta) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", line 242, in _putMetadata 12:33:20 f.write(data) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", line 161, in write 12:33:20 raise OSError(err, msg) 12:33:20 OSError: [Errno 22] Invalid argument 12:33:20 -------------------- >> begin captured logging << -------------------- 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] (MainThread) Created a persistent dict with VGTagMetadataRW backend 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) read lines (VGTagMetadataRW)=[] 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Empty metadata 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Starting transaction 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) Flushing changes 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, MDT_VERSION=3 removing= 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] (MainThread) Finished transaction 12:33:20 2016-09-28 05:32:03,261 INFO [storage.BlockVolumeArtifacts] (MainThread) Create placeholder /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 for image's volumes 12:33:20 2016-09-28 05:32:03,264 WARNING [storage.StorageDomainManifest] (MainThread) Could not find mapping for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 12:33:20 2016-09-28 05:32:03,264 DEBUG [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG c088020e-91b2-45e6-85fd-eea3fce58764 12:33:20 --------------------- >> end captured logging << --------------------- 1 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Barak Korren bkorren@redhat.com RHEV-CI Team
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/ Then I triggered the tests and they passed: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/ Thanks Nir
On 28 September 2016 at 22:00, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
It seems that the CI setup has changed, and /var/tmp is using now tempfs.
This is not compatible with vdsm tests, assuming that /var/tmp is a real file system. This is the reason we do not use /tmp.
We have lot of storage tests using directio, and directio cannot work on tempfs.
Please check the slaves and make sure /var/tmp is using file system supporting directio.
See example failure bellow.
Nir
----
12:33:20 ====================================================================== 12:33:20 ERROR: test_create_fail_creating_lease (storage_volume_artifacts_test.BlockVolumeArtifactsTests) 12:33:20 ---------------------------------------------------------------------- 12:33:20 Traceback (most recent call last): 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", line 485, in test_create_fail_creating_lease 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in assertRaises 12:33:20 callableObj(*args, **kwargs) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 391, in create 12:33:20 desc, parent) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 482, in _create_metadata 12:33:20 sc.LEGAL_VOL) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 427, in newMetadata 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 420, in createMetadata 12:33:20 cls._putMetadata(metaId, meta) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", line 242, in _putMetadata 12:33:20 f.write(data) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", line 161, in write 12:33:20 raise OSError(err, msg) 12:33:20 OSError: [Errno 22] Invalid argument 12:33:20 -------------------- >> begin captured logging << -------------------- 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] (MainThread) Created a persistent dict with VGTagMetadataRW backend 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) read lines (VGTagMetadataRW)=[] 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Empty metadata 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Starting transaction 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) Flushing changes 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, MDT_VERSION=3 removing= 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] (MainThread) Finished transaction 12:33:20 2016-09-28 05:32:03,261 INFO [storage.BlockVolumeArtifacts] (MainThread) Create placeholder /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 for image's volumes 12:33:20 2016-09-28 05:32:03,264 WARNING [storage.StorageDomainManifest] (MainThread) Could not find mapping for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 12:33:20 2016-09-28 05:32:03,264 DEBUG [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG c088020e-91b2-45e6-85fd-eea3fce58764 12:33:20 --------------------- >> end captured logging << --------------------- 1 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Barak Korren bkorren@redhat.com RHEV-CI Team
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/
Oops, wrong build. This is the failing build: http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console And this is probably the reason - using a ram disk: 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk. Attaching full console log before the job is deleted.
Then I triggered the tests and they passed: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/
Thanks Nir
On 28 September 2016 at 22:00, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
It seems that the CI setup has changed, and /var/tmp is using now tempfs.
This is not compatible with vdsm tests, assuming that /var/tmp is a real file system. This is the reason we do not use /tmp.
We have lot of storage tests using directio, and directio cannot work on tempfs.
Please check the slaves and make sure /var/tmp is using file system supporting directio.
See example failure bellow.
Nir
----
12:33:20 ====================================================================== 12:33:20 ERROR: test_create_fail_creating_lease (storage_volume_artifacts_test.BlockVolumeArtifactsTests) 12:33:20 ---------------------------------------------------------------------- 12:33:20 Traceback (most recent call last): 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", line 485, in test_create_fail_creating_lease 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in assertRaises 12:33:20 callableObj(*args, **kwargs) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 391, in create 12:33:20 desc, parent) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 482, in _create_metadata 12:33:20 sc.LEGAL_VOL) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 427, in newMetadata 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 420, in createMetadata 12:33:20 cls._putMetadata(metaId, meta) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", line 242, in _putMetadata 12:33:20 f.write(data) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", line 161, in write 12:33:20 raise OSError(err, msg) 12:33:20 OSError: [Errno 22] Invalid argument 12:33:20 -------------------- >> begin captured logging << -------------------- 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] (MainThread) Created a persistent dict with VGTagMetadataRW backend 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) read lines (VGTagMetadataRW)=[] 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Empty metadata 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Starting transaction 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) Flushing changes 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, MDT_VERSION=3 removing= 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] (MainThread) Finished transaction 12:33:20 2016-09-28 05:32:03,261 INFO [storage.BlockVolumeArtifacts] (MainThread) Create placeholder /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 for image's volumes 12:33:20 2016-09-28 05:32:03,264 WARNING [storage.StorageDomainManifest] (MainThread) Could not find mapping for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 12:33:20 2016-09-28 05:32:03,264 DEBUG [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG c088020e-91b2-45e6-85fd-eea3fce58764 12:33:20 --------------------- >> end captured logging << --------------------- 1 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Barak Korren bkorren@redhat.com RHEV-CI Team
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com>
wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets? Y.
Attaching full console log before the job is deleted.
Then I triggered the tests and they passed: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/
Thanks Nir
On 28 September 2016 at 22:00, Nir Soffer <nsoffer@redhat.com> wrote:
Hi all,
It seems that the CI setup has changed, and /var/tmp is using now
tempfs.
This is not compatible with vdsm tests, assuming that /var/tmp is a
real file
system. This is the reason we do not use /tmp.
We have lot of storage tests using directio, and directio cannot work on tempfs.
Please check the slaves and make sure /var/tmp is using file system supporting directio.
See example failure bellow.
Nir
----
12:33:20 ====================================================================== 12:33:20 ERROR: test_create_fail_creating_lease (storage_volume_artifacts_test.BlockVolumeArtifactsTests) 12:33:20
12:33:20 Traceback (most recent call last): 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py",
line 485, in test_create_fail_creating_lease 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in assertRaises 12:33:20 callableObj(*args, **kwargs) 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py",
line 391, in create 12:33:20 desc, parent) 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py",
line 482, in _create_metadata 12:33:20 sc.LEGAL_VOL) 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py",
line 427, in newMetadata 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py",
line 420, in createMetadata 12:33:20 cls._putMetadata(metaId, meta) 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py",
line 242, in _putMetadata 12:33:20 f.write(data) 12:33:20 File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py",
line 161, in write 12:33:20 raise OSError(err, msg) 12:33:20 OSError: [Errno 22] Invalid argument 12:33:20 -------------------- >> begin captured logging <<
12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] (MainThread) Created a persistent dict with VGTagMetadataRW backend 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) read lines (VGTagMetadataRW)=[] 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Empty metadata 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Starting transaction 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) Flushing changes 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, MDT_VERSION=3 removing= 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] (MainThread) Finished transaction 12:33:20 2016-09-28 05:32:03,261 INFO [storage.BlockVolumeArtifacts] (MainThread) Create placeholder
/var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7
for image's volumes 12:33:20 2016-09-28 05:32:03,264 WARNING [storage.StorageDomainManifest] (MainThread) Could not find mapping for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 12:33:20 2016-09-28 05:32:03,264 DEBUG [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG c088020e-91b2-45e6-85fd-eea3fce58764 12:33:20 --------------------- >> end captured logging <<
1 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Barak Korren bkorren@redhat.com RHEV-CI Team
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets?
I don't know, but it is easy to test - it this works the tests will work: dd if=/dev/zero of=file bs=512 count=1 oflag=direct
Evgheni, Can you try switching the current RAM drive with zram? On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets?
I don't know, but it is easy to test - it this works the tests will work:
dd if=/dev/zero of=file bs=512 count=1 oflag=direct _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hi, Indeed the proposed dd test does not work on zRAM slaves. Can we modify the job not to run on nodes with ram_disk label? The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far. Regards, Evgheni Dereveanchin ----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com>, "Evgheni Dereveanchin" <ederevea@redhat.com> Cc: "Yaniv Kaul" <ykaul@redhat.com>, "devel" <devel@ovirt.org>, "infra" <infra@ovirt.org> Sent: Thursday, 29 September, 2016 8:08:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI Evgheni, Can you try switching the current RAM drive with zram? On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets?
I don't know, but it is easy to test - it this works the tests will work:
dd if=/dev/zero of=file bs=512 count=1 oflag=direct _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Sep 29, 2016 10:28 AM, "Evgheni Dereveanchin" <ederevea@redhat.com> wrote:
Hi,
Indeed the proposed dd test does not work on zRAM slaves. Can we modify the job not to run on nodes with ram_disk label?
Are those zram based or ram based *virtio-blk* disks, or zram/ram disks within the VMs? The former should work. The latter - no idea.
The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far.
That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours. Y.
Regards, Evgheni Dereveanchin
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com>, "Evgheni Dereveanchin" <
ederevea@redhat.com>
Cc: "Yaniv Kaul" <ykaul@redhat.com>, "devel" <devel@ovirt.org>, "infra" < infra@ovirt.org> Sent: Thursday, 29 September, 2016 8:08:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Evgheni, Can you try switching the current RAM drive with zram?
On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets?
I don't know, but it is easy to test - it this works the tests will work:
dd if=/dev/zero of=file bs=512 count=1 oflag=direct _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hi Yaniv, this is a physical server with work directories created on a zRAM device, here's the patch: https://gerrit.ovirt.org/#/c/62249/2/site/ovirt_jenkins_slave/templates/prep... I'll still need to read up on this but the only slave having this class (ovirt-srv08) is now offline and should not cause issues. I tested on VM slaves and did not see errors from the dd test command you provided. Please tell me if you see errors on other nodes and I'll check what's going on but it must be something else than RAM disks. Regards, Evgheni Dereveanchin ----- Original Message ----- From: "Yaniv Kaul" <ykaul@redhat.com> To: "Evgheni Dereveanchin" <ederevea@redhat.com> Cc: "infra" <infra@ovirt.org>, "devel" <devel@ovirt.org>, "Eyal Edri" <eedri@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Thursday, 29 September, 2016 9:32:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI On Sep 29, 2016 10:28 AM, "Evgheni Dereveanchin" <ederevea@redhat.com> wrote:
Hi,
Indeed the proposed dd test does not work on zRAM slaves. Can we modify the job not to run on nodes with ram_disk label?
Are those zram based or ram based *virtio-blk* disks, or zram/ram disks within the VMs? The former should work. The latter - no idea.
The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far.
That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours. Y.
Regards, Evgheni Dereveanchin
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com>, "Evgheni Dereveanchin" <
ederevea@redhat.com>
Cc: "Yaniv Kaul" <ykaul@redhat.com>, "devel" <devel@ovirt.org>, "infra" < infra@ovirt.org> Sent: Thursday, 29 September, 2016 8:08:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Evgheni, Can you try switching the current RAM drive with zram?
On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren@redhat.com> wrote:
The CI setup did not change recently.
Great
All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go.
But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets?
I don't know, but it is easy to test - it this works the tests will work:
dd if=/dev/zero of=file bs=512 count=1 oflag=direct _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
zram does not support direct IO (tested, indeed fails). What I do is host the VMs there, though - this is working - but I'm using Lago (and not oVirt). does oVirt need direct IO for the temp disks? I thought we are doing them on the libvirt level? This is the command I use: sudo modprobe zram num_devices=1 && sudo zramctl --find --size 12G && sudo mkfs.xfs -K /dev/zram0 && sudo mount -o nobarrier /dev/zram0 /home/zram && sudo chmod 777 /home/zram And then I run lago with: ./run_suite.sh -o /home/zram basic_suite_master Y. On Thu, Sep 29, 2016 at 10:47 AM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi Yaniv,
this is a physical server with work directories created on a zRAM device, here's the patch: https://gerrit.ovirt.org/#/c/62249/2/site/ovirt_jenkins_ slave/templates/prepare-ram-disk.service.erb
I'll still need to read up on this but the only slave having this class (ovirt-srv08) is now offline and should not cause issues. I tested on VM slaves and did not see errors from the dd test command you provided.
Please tell me if you see errors on other nodes and I'll check what's going on but it must be something else than RAM disks.
Regards, Evgheni Dereveanchin
----- Original Message ----- From: "Yaniv Kaul" <ykaul@redhat.com> To: "Evgheni Dereveanchin" <ederevea@redhat.com> Cc: "infra" <infra@ovirt.org>, "devel" <devel@ovirt.org>, "Eyal Edri" < eedri@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Sent: Thursday, 29 September, 2016 9:32:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Sep 29, 2016 10:28 AM, "Evgheni Dereveanchin" <ederevea@redhat.com> wrote:
Hi,
Indeed the proposed dd test does not work on zRAM slaves. Can we modify the job not to run on nodes with ram_disk label?
Are those zram based or ram based *virtio-blk* disks, or zram/ram disks within the VMs? The former should work. The latter - no idea.
The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far.
That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours. Y.
Regards, Evgheni Dereveanchin
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "Nir Soffer" <nsoffer@redhat.com>, "Evgheni Dereveanchin" <
Cc: "Yaniv Kaul" <ykaul@redhat.com>, "devel" <devel@ovirt.org>, "infra" < infra@ovirt.org> Sent: Thursday, 29 September, 2016 8:08:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Evgheni, Can you try switching the current RAM drive with zram?
On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer@redhat.com> wrote:
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer@redhat.com>
wrote:
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren < bkorren@redhat.com> wrote: > The CI setup did not change recently.
Great
> All standard-CI jobs run inside mock (chroot) which is stored on top > of a regular FS, so they should not be affected by the slave OS at all > as far as FS settings go. > > But perhaps some slave-OS/mock-OS combination is acting strangely, so > could you be more specific and point to particular job runs that fail?
This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- x86_64/2530/
Oops, wrong build.
This is the failing build:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/1054/console
And this is probably the reason - using a ram disk:
12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx
ederevea@redhat.com> physical
integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64
We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk.
Will it work on zram? What if we configure ram based iSCSI targets?
I don't know, but it is easy to test - it this works the tests will work:
dd if=/dev/zero of=file bs=512 count=1 oflag=direct _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On 29 September 2016 at 11:08, Yaniv Kaul <ykaul@redhat.com> wrote:
zram does not support direct IO (tested, indeed fails). What I do is host the VMs there, though - this is working - but I'm using Lago (and not oVirt). does oVirt need direct IO for the temp disks? I thought we are doing them on the libvirt level?
Its not oVirt that needs those, its some specific VDSM unit tests. -- Barak Korren bkorren@redhat.com RHEV-CI Team
On Thu, Sep 29, 2016 at 11:21 AM, Barak Korren <bkorren@redhat.com> wrote:
On 29 September 2016 at 11:08, Yaniv Kaul <ykaul@redhat.com> wrote:
zram does not support direct IO (tested, indeed fails). What I do is host the VMs there, though - this is working - but I'm using Lago (and not oVirt). does oVirt need direct IO for the temp disks? I thought we are doing them on the libvirt level?
Its not oVirt that needs those, its some specific VDSM unit tests.
oVirt needs use directio when accessing shared storage. In the tests we have small integration tests that create storage domains and volume metadata using directio, or test code that used to work with directio (e.g. vdsm.storage.directio.py). Same for ovirt-imageio tests, most tests uses directio. And there are also some ioprocess tests using directio. We can use a loop device for these tests: $ truncate -s 10G /tmp/backing-file $ sudo losetup -f /tmp/backing-file --show /dev/loop0 $ sudo time dd if=/dev/zero of=/dev/loop0 bs=1M count=1024 oflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.408083 s, 2.6 GB/s 0.00user 0.10system 0:00.40elapsed 25%CPU (0avgtext+0avgdata 3128maxresident)k 0inputs+2097152outputs (0major+333minor)pagefaults 0swaps Maybe we can mount a loop device at /var/tmp in the test environment? Nir
The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far.
That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours.
What amount of RAM you had available to Linux file system cache and were there any previous runs so Linux were able to put any mock caches into the RAM cache? Besides the possible difference in disk speeds I think the second factor is this Linux fs cache that basically create an analog of RAM disk on the fly. Those two things might explain why we do not see any performance improvement from RAM drives in our case. Anton.
On Thu, Sep 29, 2016 at 12:55 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far.
That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours.
What amount of RAM you had available to Linux file system cache and were there any previous runs so Linux were able to put any mock caches into the RAM cache?
I don't do mock. And if I run everything in RAM (whether directly under /dev/shm/<somewhere> or in a zram disk), I honestly don't need the Linux system cache.
Besides the possible difference in disk speeds I think the second factor is this Linux fs cache that basically create an analog of RAM disk on the fly.
Well, theoretically, if you have enough RAM and you keep re-running, many of the data is indeed going to be cached. I'd argue that it's a better use of RAM to just run it there.
Those two things might explain why we do not see any performance improvement from RAM drives in our case.
Indeed. Y.
Anton.
Hello Yaniv. I don't do mock. And if I run everything in RAM (whether directly under
/dev/shm/<somewhere> or in a zram disk), I honestly don't need the Linux system cache.
Besides the possible difference in disk speeds I think the second factor is this Linux fs cache that basically create an analog of RAM disk on the fly.
Well, theoretically, if you have enough RAM and you keep re-running, many of the data is indeed going to be cached. I'd argue that it's a better use of RAM to just run it there.
Ok then I guess we traced down the reason. I did not do any tests to confirm this yet but I believe the "mock" is causing this effect. On our side we use mock that is at the moment essential part of the standard CI offering. Mock utilizes file system cache of the chroots and those cashes are shared between runs that along with Linux fs cache means that they are probably already were identified to be good candidates for RAM fs cache and put there. So since mock has own cache it should be trivial for Linux to effectively identify them and put into RAM and this benefits all runs that use mock. If we remove the mock than it is completely different. Anton. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
participants (6)
-
Anton Marchukov -
Barak Korren -
Evgheni Dereveanchin -
Eyal Edri -
Nir Soffer -
Yaniv Kaul