Hi all, It seems that the CI setup has changed, and /var/tmp is using now tempfs. This is not compatible with vdsm tests, assuming that /var/tmp is a real file system. This is the reason we do not use /tmp. We have lot of storage tests using directio, and directio cannot work on tempfs. Please check the slaves and make sure /var/tmp is using file system supporting directio. See example failure bellow. Nir ---- 12:33:20 ====================================================================== 12:33:20 ERROR: test_create_fail_creating_lease (storage_volume_artifacts_test.BlockVolumeArtifactsTests) 12:33:20 ---------------------------------------------------------------------- 12:33:20 Traceback (most recent call last): 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", line 485, in test_create_fail_creating_lease 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in assertRaises 12:33:20 callableObj(*args, **kwargs) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 391, in create 12:33:20 desc, parent) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", line 482, in _create_metadata 12:33:20 sc.LEGAL_VOL) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 427, in newMetadata 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", line 420, in createMetadata 12:33:20 cls._putMetadata(metaId, meta) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", line 242, in _putMetadata 12:33:20 f.write(data) 12:33:20 File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", line 161, in write 12:33:20 raise OSError(err, msg) 12:33:20 OSError: [Errno 22] Invalid argument 12:33:20 -------------------- >> begin captured logging << -------------------- 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] (MainThread) Created a persistent dict with VGTagMetadataRW backend 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) read lines (VGTagMetadataRW)=[] 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Empty metadata 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] (MainThread) Starting transaction 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) Flushing changes 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, MDT_VERSION=3 removing= 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] (MainThread) Finished transaction 12:33:20 2016-09-28 05:32:03,261 INFO [storage.BlockVolumeArtifacts] (MainThread) Create placeholder /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 for image's volumes 12:33:20 2016-09-28 05:32:03,264 WARNING [storage.StorageDomainManifest] (MainThread) Could not find mapping for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 12:33:20 2016-09-28 05:32:03,264 DEBUG [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG c088020e-91b2-45e6-85fd-eea3fce58764 12:33:20 --------------------- >> end captured logging << --------------------- 1 _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > The CI setup did not change recently. Great > All standard-CI jobs run inside mock (chroot) which is stored on top > of a regular FS, so they should not be affected by the slave OS at all > as far as FS settings go. > > But perhaps some slave-OS/mock-OS combination is acting strangely, so > could you be more specific and point to particular job runs that fail? This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/

Then I triggered the tests and they passed: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/ Thanks Nir > > On 28 September 2016 at 22:00, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> Hi all, >> >> It seems that the CI setup has changed, and /var/tmp is using now tempfs. >> >> This is not compatible with vdsm tests, assuming that /var/tmp is a real file >> system. This is the reason we do not use /tmp. >> >> We have lot of storage tests using directio, and directio cannot work on >> tempfs. >> >> Please check the slaves and make sure /var/tmp is using file system supporting >> directio. >> >> See example failure bellow. >> >> Nir >> >> ---- >> >> 12:33:20 ====================================================================== >> 12:33:20 ERROR: test_create_fail_creating_lease >> (storage_volume_artifacts_test.BlockVolumeArtifactsTests) >> 12:33:20 ---------------------------------------------------------------------- >> 12:33:20 Traceback (most recent call last): >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", >> line 485, in test_create_fail_creating_lease >> 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) >> 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in >> assertRaises >> 12:33:20 callableObj(*args, **kwargs) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", >> line 391, in create >> 12:33:20 desc, parent) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", >> line 482, in _create_metadata >> 12:33:20 sc.LEGAL_VOL) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", >> line 427, in newMetadata >> 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", >> line 420, in createMetadata >> 12:33:20 cls._putMetadata(metaId, meta) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", >> line 242, in _putMetadata >> 12:33:20 f.write(data) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", >> line 161, in write >> 12:33:20 raise OSError(err, msg) >> 12:33:20 OSError: [Errno 22] Invalid argument >> 12:33:20 -------------------- >> begin captured logging << -------------------- >> 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] >> (MainThread) Created a persistent dict with VGTagMetadataRW backend >> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >> (MainThread) read lines (VGTagMetadataRW)=[] >> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >> (MainThread) Empty metadata >> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >> (MainThread) Starting transaction >> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] >> (MainThread) Flushing changes >> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] >> (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', >> 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', >> 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', >> '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] >> 12:33:20 2016-09-28 05:32:03,257 DEBUG >> [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata >> adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, >> MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, >> MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, >> MDT_VERSION=3 removing= >> 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] >> (MainThread) Finished transaction >> 12:33:20 2016-09-28 05:32:03,261 INFO >> [storage.BlockVolumeArtifacts] (MainThread) Create placeholder >> /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 >> for image's volumes >> 12:33:20 2016-09-28 05:32:03,264 WARNING >> [storage.StorageDomainManifest] (MainThread) Could not find mapping >> for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 >> 12:33:20 2016-09-28 05:32:03,264 DEBUG >> [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG >> c088020e-91b2-45e6-85fd-eea3fce58764 >> 12:33:20 --------------------- >> end captured logging << --------------------- >> 1 >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra > > > > -- > Barak Korren > bkorren(a)redhat.com > RHEV-CI Team

On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer(a)redhat.com&gt; wrote: > > On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren(a)redhat.com&gt; > > wrote: > >> The CI setup did not change recently. > > > > Great > > > >> All standard-CI jobs run inside mock (chroot) which is stored on top > >> of a regular FS, so they should not be affected by the slave OS at all > >> as far as FS settings go. > >> > >> But perhaps some slave-OS/mock-OS combination is acting strangely, so > >> could you be more specific and point to particular job runs that fail? > > > > This jobs failed, but it was deleted (I get 404 now): > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/ > > Oops, wrong build. > > This is the failing build: > > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console > > And this is probably the reason - using a ram disk: > > 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > integ-tests ram_disk fc23) in workspace > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > We cannot run the storage tests using a ramdisk. We are creating > (tiny) volumes and storage domains and doing copies, this code cannot > work with ramdisk. Will it work on zram? What if we configure ram based iSCSI targets?

On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer(a)redhat.com&gt; wrote: >> >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren(a)redhat.com&gt; >> > wrote: >> >> The CI setup did not change recently. >> > >> > Great >> > >> >> All standard-CI jobs run inside mock (chroot) which is stored on top >> >> of a regular FS, so they should not be affected by the slave OS at all >> >> as far as FS settings go. >> >> >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so >> >> could you be more specific and point to particular job runs that fail? >> > >> > This jobs failed, but it was deleted (I get 404 now): >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- x86_64/2530/ >> >> Oops, wrong build. >> >> This is the failing build: >> >> >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/1054/console >> >> And this is probably the reason - using a ram disk: >> >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical >> integ-tests ram_disk fc23) in workspace >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >> >> We cannot run the storage tests using a ramdisk. We are creating >> (tiny) volumes and storage domains and doing copies, this code cannot >> work with ramdisk. > > Will it work on zram? > What if we configure ram based iSCSI targets? I don't know, but it is easy to test - it this works the tests will work: dd if=/dev/zero of=file bs=512 count=1 oflag=direct _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

Hi, Indeed the proposed dd test does not work on zRAM slaves. Can we modify the job not to run on nodes with ram_disk label?

The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far.

Regards, Evgheni Dereveanchin ----- Original Message ----- From: "Eyal Edri" <eedri(a)redhat.com&gt; To: "Nir Soffer" <nsoffer(a)redhat.com&gt;, "Evgheni Dereveanchin" <

Cc: "Yaniv Kaul" <ykaul(a)redhat.com&gt;, "devel" <devel(a)ovirt.org&gt;, "infra" <

Sent: Thursday, 29 September, 2016 8:08:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI Evgheni, Can you try switching the current RAM drive with zram? On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > On Sep 28, 2016 11:37 PM, "Nir Soffer" <nsoffer(a)redhat.com&gt; wrote: > >> > >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer(a)redhat.com&gt; > wrote: > >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren(a)redhat.com&gt; > >> > wrote: > >> >> The CI setup did not change recently. > >> > > >> > Great > >> > > >> >> All standard-CI jobs run inside mock (chroot) which is stored on

> >> >> of a regular FS, so they should not be affected by the slave OS at > all > >> >> as far as FS settings go. > >> >> > >> >> But perhaps some slave-OS/mock-OS combination is acting

> >> >> could you be more specific and point to particular job runs that > fail? > >> > > >> > This jobs failed, but it was deleted (I get 404 now): > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > x86_64/2530/ > >> > >> Oops, wrong build. > >> > >> This is the failing build: > >> > >> > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- > x86_64/1054/console > >> > >> And this is probably the reason - using a ram disk: > >> > >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > >> integ-tests ram_disk fc23) in workspace > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > >> > >> We cannot run the storage tests using a ramdisk. We are creating > >> (tiny) volumes and storage domains and doing copies, this code cannot > >> work with ramdisk. > > > > Will it work on zram? > > What if we configure ram based iSCSI targets? > > I don't know, but it is easy to test - it this works the tests will

> > dd if=/dev/zero of=file bs=512 count=1 oflag=direct > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

zram does not support direct IO (tested, indeed fails). What I do is host the VMs there, though - this is working - but I'm using Lago (and not oVirt). does oVirt need direct IO for the temp disks? I thought we are doing them on the libvirt level?

> The node will be offline for now until we agree on what to do. > An option is to abandon RAM disks completely as we didn't find > any performance benefits from using them so far. That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours.

/dev/shm/<somewhere> or in a zram disk), I honestly don't need the Linux system cache. > > Besides the possible difference in disk speeds I think the second factor > is this Linux fs cache that basically create an analog of RAM disk on the > fly. > Well, theoretically, if you have enough RAM and you keep re-running, many of the data is indeed going to be cached. I'd argue that it's a better use of RAM to just run it there.