[VDSM] All tests using directio fail on CI

Wed Sep 28 20:37:19 UTC 2016

On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren <bkorren at redhat.com> wrote:
>> The CI setup did not change recently.
>
> Great
>
>> All standard-CI jobs run inside mock (chroot) which is stored on top
>> of a regular FS, so they should not be affected by the slave OS at all
>> as far as FS settings go.
>>
>> But perhaps some slave-OS/mock-OS combination is acting strangely, so
>> could you be more specific and point to particular job runs that fail?
>
> This jobs failed, but it was deleted (I get 404 now):
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/

Oops, wrong build.

This is the failing build:

http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console

And this is probably the reason - using a ram disk:

12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical
integ-tests ram_disk fc23) in workspace
/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64

We cannot run the storage tests using a ramdisk. We are creating
(tiny) volumes and storage domains and doing copies, this code cannot
work with ramdisk.

Attaching full console log before the job is deleted.

>
> Then I triggered the tests and they passed:
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/
>
> Thanks
> Nir
>
>>
>> On 28 September 2016 at 22:00, Nir Soffer <nsoffer at redhat.com> wrote:
>>> Hi all,
>>>
>>> It seems that the CI setup has changed, and /var/tmp is using now tempfs.
>>>
>>> This is not compatible with vdsm tests, assuming that /var/tmp is a real file
>>> system. This is the reason we do not use /tmp.
>>>
>>> We have lot of storage tests using directio, and directio cannot work on
>>> tempfs.
>>>
>>> Please check the slaves and make sure /var/tmp is using file system supporting
>>> directio.
>>>
>>> See example failure bellow.
>>>
>>> Nir
>>>
>>> ----
>>>
>>> 12:33:20 ======================================================================
>>> 12:33:20 ERROR: test_create_fail_creating_lease
>>> (storage_volume_artifacts_test.BlockVolumeArtifactsTests)
>>> 12:33:20 ----------------------------------------------------------------------
>>> 12:33:20 Traceback (most recent call last):
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py",
>>> line 485, in test_create_fail_creating_lease
>>> 12:33:20     *BASE_PARAMS[sc.RAW_FORMAT])
>>> 12:33:20   File "/usr/lib64/python2.7/unittest/case.py", line 513, in
>>> assertRaises
>>> 12:33:20     callableObj(*args, **kwargs)
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py",
>>> line 391, in create
>>> 12:33:20     desc, parent)
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py",
>>> line 482, in _create_metadata
>>> 12:33:20     sc.LEGAL_VOL)
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py",
>>> line 427, in newMetadata
>>> 12:33:20     cls.createMetadata(metaId, meta.legacy_info())
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py",
>>> line 420, in createMetadata
>>> 12:33:20     cls._putMetadata(metaId, meta)
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py",
>>> line 242, in _putMetadata
>>> 12:33:20     f.write(data)
>>> 12:33:20   File
>>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py",
>>> line 161, in write
>>> 12:33:20     raise OSError(err, msg)
>>> 12:33:20 OSError: [Errno 22] Invalid argument
>>> 12:33:20 -------------------- >> begin captured logging << --------------------
>>> 12:33:20 2016-09-28 05:32:03,254 DEBUG   [storage.PersistentDict]
>>> (MainThread) Created a persistent dict with VGTagMetadataRW backend
>>> 12:33:20 2016-09-28 05:32:03,255 DEBUG   [storage.PersistentDict]
>>> (MainThread) read lines (VGTagMetadataRW)=[]
>>> 12:33:20 2016-09-28 05:32:03,255 DEBUG   [storage.PersistentDict]
>>> (MainThread) Empty metadata
>>> 12:33:20 2016-09-28 05:32:03,255 DEBUG   [storage.PersistentDict]
>>> (MainThread) Starting transaction
>>> 12:33:20 2016-09-28 05:32:03,256 DEBUG   [storage.PersistentDict]
>>> (MainThread) Flushing changes
>>> 12:33:20 2016-09-28 05:32:03,256 DEBUG   [storage.PersistentDict]
>>> (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data',
>>> 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d',
>>> 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3',
>>> '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139']
>>> 12:33:20 2016-09-28 05:32:03,257 DEBUG
>>> [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata
>>> adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d,
>>> MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139,
>>> MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data,
>>> MDT_VERSION=3 removing=
>>> 12:33:20 2016-09-28 05:32:03,257 DEBUG   [storage.PersistentDict]
>>> (MainThread) Finished transaction
>>> 12:33:20 2016-09-28 05:32:03,261 INFO
>>> [storage.BlockVolumeArtifacts] (MainThread) Create placeholder
>>> /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7
>>> for image's volumes
>>> 12:33:20 2016-09-28 05:32:03,264 WARNING
>>> [storage.StorageDomainManifest] (MainThread) Could not find mapping
>>> for lv c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3
>>> 12:33:20 2016-09-28 05:32:03,264 DEBUG
>>> [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG
>>> c088020e-91b2-45e6-85fd-eea3fce58764
>>> 12:33:20 --------------------- >> end captured logging << ---------------------
>>> 1
>>> _______________________________________________
>>> Infra mailing list
>>> Infra at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>>
>> --
>> Barak Korren
>> bkorren at redhat.com
>> RHEV-CI Team
-------------- next part --------------
A non-text attachment was scrubbed...
Name: console.log.xz
Type: application/x-xz
Size: 61132 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20160928/a42d1c2a/attachment.xz>