Hi all, On Sat, Feb 15, 2020 at 8:04 AM <jenkins@jenkins.phx.ovirt.org> wrote:

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.3/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.3/35...

Can someone from storage please have a look, why this failed? What I managed to see so far: 1. hosted-engine deploy ansible task "Add HE disks" failed. This task creates several different disks, only one failed: "Hosted-Engine sanlock disk". https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.3/35... 2. engine.log has: 2020-02-14 23:00:04,010-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (default task-1) [afc10fcf-2a07-4f6f-aaa3-dce5f68b8390] START, CreateImageVDSCommand( CreateImageVDSCommandParameters:{storagePoolId='ff474994-4fa4-11ea-812d-5452c0a8c863', ignoreFailoverLimit='false', storageDomainId='b91bd467-8b60-4be6-b909-74ad46a4e883', imageGroupId='fbd0cd90-966d-43a0-bed4-912a0082b9b1', imageSizeInBytes='1073741824', volumeFormat='RAW', newImageId='58760158-3bbe-4963-bff9-5ebb74b1631c', imageType='Preallocated', newImageDescription='{"DiskAlias":"he_sanlock","DiskDescription":"Hosted-Engine sanlock disk"}', imageInitialSizeInBytes='0'}), log id: a29934f and later: 2020-02-14 23:00:15,868-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lago-he-basic-iscsi-suite-4-3-host-0 command HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u"Volume creation 58760158-3bbe-4963-bff9-5ebb74b1631c failed: (104, 'Sanlock resource write failure', 'Connection reset by peer')",) https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.3/35... 3. vdsm.log has: 2020-02-14 23:00:04,020-0500 INFO (jsonrpc/4) [vdsm.api] START createVolume(sdUUID=u'b91bd467-8b60-4be6-b909-74ad46a4e883', spUUID=u'ff474994-4fa4-11ea-812d-5452c0a8c863', imgUUID=u'fbd0cd90-966d-43a0-bed4-912a0082b9b1', size=u'1073741824', volFormat=5, preallocate=1, diskType=u'HESD', volUUID=u'58760158-3bbe-4963-bff9-5ebb74b1631c', desc=u'{"DiskAlias":"he_sanlock","DiskDescription":"Hosted-Engine sanlock disk"}', srcImgUUID=u'00000000-0000-0000-0000-000000000000', srcVolUUID=u'00000000-0000-0000-0000-000000000000', initialSize=None) from=::ffff:192.168.222.76,56570, flow_id=afc10fcf-2a07-4f6f-aaa3-dce5f68b8390, task_id=1d0334c0-469f-474a-b310-8ecfd973f8fc (api:48) and eventually: 2020-02-14 23:00:06,535-0500 ERROR (tasks/3) [storage.Volume] Unexpected error (volume:1278) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1272, in create cls.newVolumeLease(metaId, sdUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1555, in newVolumeLease return cls.manifestClass.newVolumeLease(metaId, sdUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 296, in newVolumeLease sd.create_volume_lease(slot, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 976, in create_volume_lease sector=self.block_size) SanlockException: (104, 'Sanlock resource write failure', 'Connection reset by peer') 2020-02-14 23:00:06,536-0500 ERROR (tasks/3) [storage.TaskManager.Task] (Task='1d0334c0-469f-474a-b310-8ecfd973f8fc') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1940, in createVolume initial_size=initialSize) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 930, in createVolume initial_size=initial_size) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1280, in create (volUUID, e)) VolumeCreationError: Error creating a new volume: (u"Volume creation 58760158-3bbe-4963-bff9-5ebb74b1631c failed: (104, 'Sanlock resource write failure', 'Connection reset by peer')",) https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-4.3/35... Thanks,

Build Number: 350 Build Status: Failure Triggered By: Started by timer

------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #350 [Anton Marchukov] Added "sar" system resources collection on VMs.

----------------- Failed Tests: ----------------- No tests ran.

-- Didi