New subject: Ovirt: Upload Disk Snapshots problems with iSCSI storage and Java SDK

Wednesday, 8 May 2019

Description of the problem:
I am not able to restore (upload) a VM when its disk (Either Thin provisioning or Thick
provisioned) is located in an iSCSI storage and this VM has at least one snapshot larger
than 1Gb. I am using REST API through Java SDK. 
I am using the same upload procedure described here:
https://ovirt.org/develop/release-management/features/storage/backup-rest...,
of course, making little adjustments to disk format (COW, Sparse.. Depending on the type
of the VM to be restored).
When I try restore any VM with this conditions, all snapshot disks with a size smaller
than 1Gb are uploaded correctly. But when I am uploading a snapshot disk with a size
bigger than 1Gb, the following error ocurrs:

The error triggered from Java SDK is: "The server response was 403 in the range
request {bytes 1073741824-1207959551/4831838208}"
In the host's imageio log: "2019-05-07 14:26:30,253 WARNING (Thread-71960) [web]
ERROR [172.19.33.146] PUT /images/8d353735-0a29-463c-b772-ec37f451e2e9 [403] Requested
range out of allowed range [request=0.000488]"

For example (real scenario), I have these snapshot disks from a vm:
2.1G	DiskSnap_47c4e157-edd8-4bfc-b838-4d57ac8396bd_33b3e74a-cc6b-474a-97cc-960891716110_0.img
-> First snap chain -> OK upload
1.1G	DiskSnap_47c4e157-edd8-4bfc-b838-4d57ac8396bd_9349e4dd-85b3-4145-8674-bb3f39546020_1.img
-> Ok upload
1.1G	DiskSnap_47c4e157-edd8-4bfc-b838-4d57ac8396bd_4d10cb58-9f52-4d64-b6e9-ead4bed4c6a6_2.img
-> Ok upload
4.6G	DiskSnap_47c4e157-edd8-4bfc-b838-4d57ac8396bd_429cbe6d-1148-4a84-861a-56ad6902859e_3.img
-> Last snap chain -> Fail upload with previous error
[Qemu-img info in attached file: qemu-imgs-iSCSI.txt]

In the restore process, when I create the disk(In this case), I put the following values:
Disk: {
	Name:T_iSCSI_Thin_Disk1_restore,
	Id:null,
	Interface:VIRTIO_SCSI,
	Format:COW,
	WipeAfterDelete: false,
	Shareable: false,
	Sparse: true,
	Boot:true,
	Activo:true,
	Sizes:{
		Initial:4294967296,
		Actual:null,
		Total:null,
		Provisioned: 4294967296
	}
}

And the disk created is:
Disk: {
	Name:T_iSCSI_Thin_Disk1_restore,
	Id:aac28ff3-abdb-4bf9-beed-c08e7e19b0ba,
	Image:c8654d05-a18c-47bd-ab0b-f4d746e23efb,
	Format:COW,
	WipeAfterDelete: false,
	Shareable: false,
	Sparse: true,
	Sizes:{
		Initial:null,
		Actual:0,
		Total:0,
		Provisioned: 4294967296
	}
}

Therefore, the first upload can finish correctly. However, when I create snapshots with
this disk, the snapshot disks have these parameters:
DiskSnapshot: {
	Id: 29dd6a18-c17c-4938-be67-d7af6de713ec,
	Disk:aac28ff3-abdb-4bf9-beed-c08e7e19b0ba,
	Snapshot:ceb2f29e-1ae8-441c-a5a7-836308cfeb8d,
	Sizes:{
		Actual:1073741824
		Provisioned:4294967296
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: 69620911-d490-4872-b353-ba29a762ea3e,
	Disk:aac28ff3-abdb-4bf9-beed-c08e7e19b0ba,
	Snapshot:c35423d5-ddd6-46f2-a5d7-0080941c3f30,
	Sizes:{
		Actual:1073741824
		Provisioned:4294967296
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: c8654d05-a18c-47bd-ab0b-f4d746e23efb,
	Disk:aac28ff3-abdb-4bf9-beed-c08e7e19b0ba,
	Snapshot:fc135efd-a67b-4eb3-bfdb-9090aa3b267e,
	Sizes:{
		Actual:4831838208
		Provisioned:4294967296
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: e3389916-4e80-4593-a116-3484f295ff7f,
	Disk:aac28ff3-abdb-4bf9-beed-c08e7e19b0ba,
	Snapshot:74aff5e7-e1c1-4f51-ae31-ea44caa180ef,
	Sizes:{
		Actual:1073741824
		Provisioned:4294967296
		Total: 0
		Initial: null
	}
}

As we can observe with these values, the actual size of snapshot disks indicates 1Gb, and
when I try upload a snapshot disk (Bigger than 1.1Gb), the system doesn't allow to
complete the upload, launching the mentionated previous error.

However, using NFS storage, the previous error doesn't ocurr. The system allows me
upload disks bigger than 1 Gb, and when the upload finishes, the system refreshs the
actualSize to actual value.

For example (real scenario), I have these snapshot disks from a vm:
3.1G	DiskSnap_985a02a0-0752-420a-a2fb-a3e89d1ac3bc_3dffb5f4-3f9c-410c-8b97-133922721f9a_0.img
-> First snap chain -> OK upload
22M	 
DiskSnap_985a02a0-0752-420a-a2fb-a3e89d1ac3bc_d3307b3c-f930-4faa-90cb-6aedd38bea93_1.img
-> Ok upload
31M	 
DiskSnap_985a02a0-0752-420a-a2fb-a3e89d1ac3bc_aa8648dd-d6c0-4d9d-8373-6764e806359f_2.img
-> Ok upload
1.3G	DiskSnap_985a02a0-0752-420a-a2fb-a3e89d1ac3bc_2cf31d95-3b86-4d97-b39b-672d0066f503_3.img
-> Last snap chain -> OK upload
[Qemu-img info in attached file: qemu-imgs-NFS.txt]

The restore with NFS storage is the same process but I change the format disk and sparse.
With following values:

Disk: {
	Name:985a02a0-0752-420a-a2fb-a3e89d1ac3bc_restore,
	Id:null,
	Interface:VIRTIO_SCSI,
	Format:RAW,
	WipeAfterDelete: false,
	Shareable: false,
	Sparse: true,
	Boot:true,
	Activo:true,
	Sizes:{
		Initial:3221225472,
		Actual:null,
		Total:null,
		Provisioned: 3221225472
	}
}

And the disk created was:
Disk: {
	Name:985a02a0-0752-420a-a2fb-a3e89d1ac3bc_restore,
	Id:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Image:9c80aead-88de-48f9-be7e-0fde917e7f55,
	Format:RAW,
	WipeAfterDelete: false,
	Shareable: false,
	Sparse: true,
	Sizes:{
		Initial:null,
		Actual:0,
		Total:0,
		Provisioned: 3221225472
	}
}

When I create snapshots with this disk, the snapshot disks have these parameters:

DiskSnapshot: {
	Id: 87c4cb08-502a-4937-bbba-6d455395e990,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:dd257190-ef50-4990-a61f-40d84e2b08e9,
	Sizes:{
		Actual:200704
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: e7d6f1fb-ab37-45f8-8e23-5964ed194581,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:b9f3f3d3-710e-4ce2-b05f-563227f5ec04,
	Sizes:{
		Actual:200704
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: 8ef6f502-d00f-4fa5-b407-2cdc0e876045,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:274994de-5c2b-4b38-965f-4cadea3e0db3,
	Sizes:{
		Actual:200704
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: 9c80aead-88de-48f9-be7e-0fde917e7f55,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:f1d7382d-4427-4e21-9544-1b3cd85f23ae,
	Sizes:{
		Actual:0
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}

As we can observe with these values, the actual size of snapshot disks indicates 196Kb,
and when I try upload any snapshot disk (Even more 1.1Gb), 
the system allows upload all content disk and when it is finalished, it changes actual and
total size. When the restore ends, I obtain these values:

Disk: {
	Name:985a02a0-0752-420a-a2fb-a3e89d1ac3bc_restoreBacula,
	Id:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Image:8ef6f502-d00f-4fa5-b407-2cdc0e876045,
	Format:COW,
	WipeAfterDelete: false,
	Shareable: false,
	Status: OK,
	Sparse: true,
	Sizes:{
		Initial:null,
		Actual:1318850560,
		Total:4593303552,
		Provisioned: 3221225472
	}
}

DiskSnapshot: {
	Id: 87c4cb08-502a-4937-bbba-6d455395e990,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:dd257190-ef50-4990-a61f-40d84e2b08e9,
	Sizes:{
		Actual:22282240
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: e7d6f1fb-ab37-45f8-8e23-5964ed194581,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:b9f3f3d3-710e-4ce2-b05f-563227f5ec04,
	Sizes:{
		Actual:32374784
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}
DiskSnapshot: {
	Id: 9c80aead-88de-48f9-be7e-0fde917e7f55,
	Disk:f2a89f5a-d8ac-4ea4-b17c-591cd86855e9,
	Snapshot:f1d7382d-4427-4e21-9544-1b3cd85f23ae,
	Sizes:{
		Actual:3219795968
		Provisioned:3221225472
		Total: 0
		Initial: null
	}
}

There is a disk snapshot less since at the end of the restoration I merge the last
snapshot. But the values showed changes in actual Size in the DiskSnapshot.

Could you help me, please? Is there a different procedure to the one described in
https://ovirt.org/develop/release-management/features/storage/backup-rest...
to work with iSCSI storage or is this simply a bug?

How reproducible:
Backup and restore VM with snapshots in an iSCSI storage, using Java SDK.

Steps to Reproduce:
1.- Follow steps indicated in
https://ovirt.org/develop/release-management/features/storage/backup-rest...
to backup a VM allocated iSCSI Storage and containing snapshots bigger than 1.1GB
2.- Try to restore it using the instructions of that documment

Actual results:
Upload of images of disk snapshots bigger than 1.1Gb do not suceed ant the restore process
fails.

Expected results:
Restored vm

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Ovirt: Upload Disk Snapshots problems with iSCSI storage and Java SDK