6:54 a.m.
--_000_28E0F4543693485EA391F19737F89D56acroniscom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
SGVsbG8sDQoNCkZpcnN0IG9mIGFsbCBJIHdvdWxkIGxpa2UgdG8gYXNrIGlmIGFueWJvZHkgaGFz
IGFuIGV4cGVyaWVuY2Ugd2l0aCB1c2luZyBNaWNyb3NvZnQgTkZTIHNlcnZlciBhcyBhIHN0b3Jh
Z2UgZG9tYWluLg0KDQpUaGUgbWFpbiBpc3N1ZSB3aXRoIE1TIE5GUyBpcyBOVEZTIDopIE5URlMg
ZG9lc24ndCBzdXBwb3J0IHNwYXJzZSBmaWxlcy4gVGVjaG5pY2FsbHkgaXQncyBwb3NzaWJsZSBi
eSBlbmFibGluZyBOVEZTIGNvbXByZXNzaW9uIGJ1dCAgaXQgaGFzIGJhZCBwZXJmb3JtYW5jZSBv
biBodWdlIGZpbGVzIHdoaWNoIGlzIG91ciBjYXNlLiBBbHNvIHRoZXJlIGlzIG5vIG9wdGlvbiBp
biBvVmlydCB3ZWIgaW50ZXJmYWNlIHRvIHVzZSBDT1cgZm9ybWF0IG9uIE5GUyBzdG9yYWdlIGRv
bWFpbnMuDQoNClNpbmNlIGl0IGxvb2tzIGxpa2Ugb1ZpcnQgZG9lc24ndCBzdXBwb3J0IE1TIE5G
UywgSSBkZWNpZGVkIHRvIG1pZ3JhdGUgYWxsIG15IFZNcyBvdXQgb2YgTVMgTkZTIHRvIGFub3Ro
ZXIgc3RvcmFnZS4gQW5kIEkgaGl0IGEgYnVnLiBMaXZlIHN0b3JhZ2UgbWlncmF0aW9uIHNpbGVu
dGx5IGNvcnJ1cHRzIGRhdGEgaWYgeW91IG1pZ3JhdGUgYSBkaXNrIGZyb20gTVMgTkZTIHN0b3Jh
Z2UgZG9tYWluLiBTbyBpZiB5b3Ugc2h1dGRvd24ganVzdCBtaWdyYXRlZCBWTSBhbmQgY2hlY2sg
ZmlsZXN5c3RlbSB5b3UgZmluZCB0aGF0IGl0IGhhcyBhIGxvdCBvZiB1bnJlY292ZXJhYmxlIGVy
cm9ycy4NCg0KVGhlcmUgYXJlIHRoZSBmb2xsb3dpbmcgc3ltcHRvbXM6DQoxLiBJdCBjb3JydXB0
cyBkYXRhIGlmIHlvdSBtaWdyYXRlIGEgZGlzayBmcm9tIE1TIE5GUyB0byBMaW51eCBORlMNCjIu
IEl0IGNvcnJ1cHRzIGRhdGEgaWYgeW91IG1pZ3JhdGUgYSBkaXNrIGZyb20gTVMgTkZTIHRvIGlT
Q1NJDQozLiBUaGVyZSBpcyBubyBjb3JydXB0aW9uIGlmIHlvdSBtaWdyYXRlIGZyb20gTGludXgg
TkZTIHRvIGlTQ1NJIGFuZCB2aWNlIHZlcnNhLg0KNC4gVGhlcmUgaXMgbm8gY29ycnVwdGlvbiBp
ZiB5b3UgbWlncmF0ZSBmcm9tIGFueXdoZXJlIHRvIE1TIE5GUy4NCjUuIERhdGEgY29ycnVwdGlv
biBoYXBwZW5zIGFmdGVyICdBdXRvLWdlbmVyYXRlZCBmb3IgTGl2ZSBTdG9yYWdlIE1pZ3JhdGlv
bicgc25hcHNob3QuIFNvIGlmIHlvdSByb2xsYmFjayB0aGUgc25hcHNob3QsIHlvdSBjb3VsZCBz
ZWUgYWJzb2x1dGVseSBjbGVhbiBmaWxlc3lzdGVtLg0KNi4gSXQgZG9lc24ndCBkZXBlbmQgb24g
U1BNLiBTbyBpdCBjb3JydXB0cyBkYXRhIGlmIFNQTSBpcyBvbiB0aGUgc2FtZSBob3N0LCBvciBh
bm90aGVyLg0KNy4gVGhlcmUgYXJlIG5vIGVycm9yIG1lc3NhZ2VzIGluIHZkc20vcWVtdS9zeXN0
ZW0gbG9ncy4NCg0KWWVzLCBvZiBjb3Vyc2UgSSBjb3VsZCBtaWdyYXRlIGZyb20gTVMgTkZTIHdp
dGggZG93bnRpbWUg4oCTIGl0J3Mgbm90IGFuIGlzc3VlLiBUaGUgaXNzdWUgaXMgdGhhdCBvVmly
dCBkb2VzIHNpbGVudGx5IGNvcnJ1cHQgZGF0YSB1bmRlciBzb21lIGNpcmN1bXN0YW5jZXMuDQoN
CkNvdWxkIHlvdSBwbGVhc2UgaGVscCBtZSB0byB1bmRlcnN0YW5kIHRoZSByZWFzb24gb2YgZGF0
YSBjb3JydXB0aW9uPw0KDQp2ZHNtLTQuMTcuMTMtMS5lbDcubm9hcmNoDQpxZW11LWltZy1ldi0y
LjMuMC0zMS5lbDdfMi40LjEueDg2XzY0DQpsaWJ2aXJ0LWRhZW1vbi0xLjIuMTctMTMuZWw3XzIu
Mi54ODZfNjQNCm92aXJ0LWVuZ2luZS1iYWNrZW5kLTMuNi4xLjMtMS5lbDcuY2VudG9zLm5vYXJj
aA0KDQpUaGFuayB5b3UNCg0KDQo=
--_000_28E0F4543693485EA391F19737F89D56acroniscom_
Content-Type: text/html; charset="utf-8"
Content-ID: <33D4DFB91E2C1549AAFC1003C4B300DA(a)acronis.com>
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5IHN0eWxlPSJ3b3JkLXdy
YXA6IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJy
ZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiPg0KPGRpdiBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAw
KTsgZm9udC1mYW1pbHk6IENhbGlicmksIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsiPg0K
SGVsbG8sPC9kaXY+DQo8ZGl2IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWls
eTogQ2FsaWJyaSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+DQo8YnI+DQo8L2Rpdj4N
CjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBz
YW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7Ij4NCkZpcnN0IG9mIGFsbCBJIHdvdWxkIGxpa2Ug
dG8gYXNrIGlmIGFueWJvZHkgaGFzIGFuIGV4cGVyaWVuY2Ugd2l0aCB1c2luZyBNaWNyb3NvZnQg
TkZTIHNlcnZlciBhcyBhIHN0b3JhZ2UgZG9tYWluLjwvZGl2Pg0KPGRpdiBzdHlsZT0iY29sb3I6
IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IENhbGlicmksIHNhbnMtc2VyaWY7IGZvbnQtc2l6
ZTogMTRweDsiPg0KPGJyPg0KPC9kaXY+DQo8ZGl2IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDAp
OyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+DQpU
aGUgbWFpbiBpc3N1ZSB3aXRoIE1TIE5GUyBpcyBOVEZTIDopIE5URlMgZG9lc24ndCBzdXBwb3J0
IHNwYXJzZSBmaWxlcy4gVGVjaG5pY2FsbHkgaXQncyBwb3NzaWJsZSBieSBlbmFibGluZyBOVEZT
IGNvbXByZXNzaW9uIGJ1dCAmbmJzcDtpdCBoYXMgYmFkIHBlcmZvcm1hbmNlIG9uIGh1Z2UgZmls
ZXMgd2hpY2ggaXMgb3VyIGNhc2UuIEFsc28gdGhlcmUgaXMgbm8gb3B0aW9uIGluIG9WaXJ0IHdl
YiBpbnRlcmZhY2UgdG8gdXNlIENPVyBmb3JtYXQgb24NCiBORlMgc3RvcmFnZSBkb21haW5zLjwv
ZGl2Pg0KPGRpdiBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IENhbGli
cmksIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsiPg0KPGJyPg0KPC9kaXY+DQo8ZGl2IHN0
eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJp
ZjsgZm9udC1zaXplOiAxNHB4OyI+DQpTaW5jZSBpdCBsb29rcyBsaWtlIG9WaXJ0IGRvZXNuJ3Qg
c3VwcG9ydCBNUyBORlMsIEkgZGVjaWRlZCB0byBtaWdyYXRlIGFsbCBteSBWTXMgb3V0IG9mIE1T
IE5GUyB0byBhbm90aGVyIHN0b3JhZ2UuIEFuZCBJIGhpdCBhIGJ1Zy4gTGl2ZSBzdG9yYWdlIG1p
Z3JhdGlvbg0KPGI+c2lsZW50bHk8L2I+Jm5ic3A7PGI+Y29ycnVwdHM8L2I+IDxiPmRhdGE8L2I+
IGlmIHlvdSBtaWdyYXRlIGEgZGlzayBmcm9tIE1TIE5GUyBzdG9yYWdlIGRvbWFpbi4gU28gaWYg
eW91IHNodXRkb3duIGp1c3QgbWlncmF0ZWQgVk0gYW5kIGNoZWNrIGZpbGVzeXN0ZW0geW91IGZp
bmQgdGhhdCBpdCBoYXMgYSBsb3Qgb2YgdW5yZWNvdmVyYWJsZSBlcnJvcnMuJm5ic3A7PC9kaXY+
DQo8ZGl2IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogQ2FsaWJyaSwg
c2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+DQo8YnI+DQo8L2Rpdj4NCjxkaXYgc3R5bGU9
ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyBm
b250LXNpemU6IDE0cHg7Ij4NClRoZXJlIGFyZSB0aGUgZm9sbG93aW5nIHN5bXB0b21zOjwvZGl2
Pg0KPGRpdiBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IENhbGlicmks
IHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsiPg0KMS4gSXQgY29ycnVwdHMgZGF0YSBpZiB5
b3UgbWlncmF0ZSBhIGRpc2sgZnJvbSBNUyBORlMgdG8gTGludXggTkZTPC9kaXY+DQo8ZGl2IHN0
eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJp
ZjsgZm9udC1zaXplOiAxNHB4OyI+DQoyLiBJdCBjb3JydXB0cyBkYXRhIGlmIHlvdSBtaWdyYXRl
IGEgZGlzayBmcm9tIE1TIE5GUyB0byBpU0NTSTwvZGl2Pg0KPGRpdiBzdHlsZT0iY29sb3I6IHJn
YigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IENhbGlicmksIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTog
MTRweDsiPg0KMy4gVGhlcmUgaXMgbm8gY29ycnVwdGlvbiBpZiB5b3UgbWlncmF0ZSBmcm9tIExp
bnV4IE5GUyB0byBpU0NTSSBhbmQgdmljZSB2ZXJzYS48L2Rpdj4NCjxkaXYgc3R5bGU9ImNvbG9y
OiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyBmb250LXNp
emU6IDE0cHg7Ij4NCjQuIFRoZXJlIGlzIG5vIGNvcnJ1cHRpb24gaWYgeW91IG1pZ3JhdGUgZnJv
bSBhbnl3aGVyZSB0byBNUyBORlMuPC9kaXY+DQo8ZGl2IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAs
IDApOyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+
DQo1LiBEYXRhIGNvcnJ1cHRpb24gaGFwcGVucyBhZnRlciAnQXV0by1nZW5lcmF0ZWQgZm9yIExp
dmUgU3RvcmFnZSBNaWdyYXRpb24nIHNuYXBzaG90LiBTbyBpZiB5b3Ugcm9sbGJhY2sgdGhlIHNu
YXBzaG90LCB5b3UgY291bGQgc2VlIGFic29sdXRlbHkgY2xlYW4gZmlsZXN5c3RlbS48L2Rpdj4N
CjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBz
YW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7Ij4NCjYuIEl0IGRvZXNuJ3QgZGVwZW5kIG9uIFNQ
TS4gU28gaXQgY29ycnVwdHMgZGF0YSBpZiBTUE0gaXMgb24gdGhlIHNhbWUgaG9zdCwgb3IgYW5v
dGhlci48L2Rpdj4NCjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5
OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7Ij4NCjcuIFRoZXJlIGFyZSBu
byBlcnJvciBtZXNzYWdlcyBpbiB2ZHNtL3FlbXUvc3lzdGVtIGxvZ3MuPC9kaXY+DQo8ZGl2IHN0
eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJp
ZjsgZm9udC1zaXplOiAxNHB4OyI+DQo8YnI+DQo8L2Rpdj4NCjxkaXYgc3R5bGU9ImNvbG9yOiBy
Z2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyBmb250LXNpemU6
IDE0cHg7Ij4NClllcywgb2YgY291cnNlIEkgY291bGQgbWlncmF0ZSBmcm9tIE1TIE5GUyB3aXRo
IGRvd250aW1lIOKAkyBpdCdzIG5vdCBhbiBpc3N1ZS4gVGhlIGlzc3VlIGlzIHRoYXQgb1ZpcnQg
ZG9lcyZuYnNwO3NpbGVudGx5IGNvcnJ1cHQgZGF0YSB1bmRlciBzb21lIGNpcmN1bXN0YW5jZXMu
PC9kaXY+DQo8ZGl2IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogQ2Fs
aWJyaSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+DQo8YnI+DQo8L2Rpdj4NCjxkaXYg
c3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBzYW5zLXNl
cmlmOyBmb250LXNpemU6IDE0cHg7Ij4NCkNvdWxkIHlvdSBwbGVhc2UgaGVscCBtZSB0byB1bmRl
cnN0YW5kIHRoZSByZWFzb24gb2YgZGF0YSBjb3JydXB0aW9uPzwvZGl2Pg0KPGRpdiBzdHlsZT0i
Y29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IENhbGlicmksIHNhbnMtc2VyaWY7IGZv
bnQtc2l6ZTogMTRweDsiPg0KPGJyPg0KPC9kaXY+DQo8ZGl2Pjxmb250IGZhY2U9IkNhbGlicmks
c2Fucy1zZXJpZiI+dmRzbS00LjE3LjEzLTEuZWw3Lm5vYXJjaDwvZm9udD48L2Rpdj4NCjxkaXY+
PGZvbnQgZmFjZT0iQ2FsaWJyaSxzYW5zLXNlcmlmIj5xZW11LWltZy1ldi0yLjMuMC0zMS5lbDdf
Mi40LjEueDg2XzY0PC9mb250PjwvZGl2Pg0KPGRpdj48Zm9udCBmYWNlPSJDYWxpYnJpLHNhbnMt
c2VyaWYiPmxpYnZpcnQtZGFlbW9uLTEuMi4xNy0xMy5lbDdfMi4yLng4Nl82NDwvZm9udD48L2Rp
dj4NCjxkaXY+PGZvbnQgZmFjZT0iQ2FsaWJyaSxzYW5zLXNlcmlmIj5vdmlydC1lbmdpbmUtYmFj
a2VuZC0zLjYuMS4zLTEuZWw3LmNlbnRvcy5ub2FyY2g8L2ZvbnQ+PC9kaXY+DQo8ZGl2Pjxmb250
IGZhY2U9IkNhbGlicmksc2Fucy1zZXJpZiI+PGJyPg0KPC9mb250PjwvZGl2Pg0KPGRpdiBzdHls
ZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IENhbGlicmksIHNhbnMtc2VyaWY7
IGZvbnQtc2l6ZTogMTRweDsiPg0KVGhhbmsgeW91PC9kaXY+DQo8ZGl2IHN0eWxlPSJjb2xvcjog
cmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJpZjsgZm9udC1zaXpl
OiAxNHB4OyI+DQo8YnI+DQo8L2Rpdj4NCjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7
IGZvbnQtZmFtaWx5OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7Ij4NCjxi
cj4NCjwvZGl2Pg0KPGRpdiBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6
IENhbGlicmksIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsiPg0KPGRpdiBpZD0iTUFDX09V
VExPT0tfU0lHTkFUVVJFIj48L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K
--_000_28E0F4543693485EA391F19737F89D56acroniscom_--
7:14 a.m.
inline
On Thu, Jan 21, 2016 at 7:54 AM, Pavel Gashev <Pax(a)acronis.com> wrote:
Hello,
First of all I would like to ask if anybody has an experience with using
Microsoft NFS server as a storage domain.
I have used one as an ISO domain for years. It wasn't great, but it was
good enough. Never a data domain though
The main issue with MS NFS is NTFS :) NTFS doesn't support sparse
files.
Technically it's possible by enabling NTFS compression but it has bad
performance on huge files which is our case. Also there is no option in
oVirt web interface to use COW format on NFS storage domains.
Since it looks like oVirt doesn't support MS NFS, I decided to migrate all
my VMs out of MS NFS to another storage. And I hit a bug. Live storage
migration *silently* *corrupts* *data* if you migrate a disk from MS NFS
storage domain. So if you shutdown just migrated VM and check filesystem
you find that it has a lot of unrecoverable errors.
There are the following symptoms:
1. It corrupts data if you migrate a disk from MS NFS to Linux NFS
2. It corrupts data if you migrate a disk from MS NFS to iSCSI
3. There is no corruption if you migrate from Linux NFS to iSCSI and vice
versa.
4. There is no corruption if you migrate from anywhere to MS NFS.
5. Data corruption happens after 'Auto-generated for Live Storage
Migration' snapshot. So if you rollback the snapshot, you could see
absolutely clean filesystem.
6. It doesn't depend on SPM. So it corrupts data if SPM is on the same
host, or another.
7. There are no error messages in vdsm/qemu/system logs.
Yes, of course I could migrate from MS NFS with downtime – it's not an
issue. The issue is that oVirt does silently corrupt data under some
circumstances.
Could you please help me to understand the reason of data corruption?
vdsm-4.17.13-1.el7.noarch
qemu-img-ev-2.3.0-31.el7_2.4.1.x86_64
libvirt-daemon-1.2.17-13.el7_2.2.x86_64
ovirt-engine-backend-3.6.1.3-1.el7.centos.noarch
Thank you
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:42 p.m.
On Thu, Jan 21, 2016 at 2:54 PM, Pavel Gashev <Pax(a)acronis.com> wrote:
Hello,
First of all I would like to ask if anybody has an experience with using
Microsoft NFS server as a storage domain.
The main issue with MS NFS is NTFS :) NTFS doesn't support sparse files.
Technically it's possible by enabling NTFS compression but it has bad
performance on huge files which is our case. Also there is no option in
oVirt web interface to use COW format on NFS storage domains.
You can
1. create a small disk (1G)
2. create a snapshot
3. extend the disk go the final size
And you have nfs with cow format. The performance difference with one snapshot
should be small.
Since it looks like oVirt doesn't support MS NFS, I decided to
migrate all
my VMs out of MS NFS to another storage. And I hit a bug. Live storage
migration silently corrupts data if you migrate a disk from MS NFS storage
domain. So if you shutdown just migrated VM and check filesystem you find
that it has a lot of unrecoverable errors.
There are the following symptoms:
1. It corrupts data if you migrate a disk from MS NFS to Linux NFS
2. It corrupts data if you migrate a disk from MS NFS to iSCSI
3. There is no corruption if you migrate from Linux NFS to iSCSI and vice
versa.
4. There is no corruption if you migrate from anywhere to MS NFS.
5. Data corruption happens after 'Auto-generated for Live Storage Migration'
snapshot. So if you rollback the snapshot, you could see absolutely clean
filesystem.
Can you try to create a live-snapshot on MS NFS? It seems that this is the
issue, not live storage migration.
Do you have qemu-guest-agent on the vm? Without qemu-guest-agent, file
systems on the guest will no be freezed during the snapshot, which may cause
inconsistent snapshot.
Can you reproduce this with virt-manager, or by creating a vm and taking
a snapshot using virsh?
6. It doesn't depend on SPM. So it corrupts data if SPM is on the
same host,
or another.
7. There are no error messages in vdsm/qemu/system logs.
Yes, of course I could migrate from MS NFS with downtime – it's not an
issue. The issue is that oVirt does silently corrupt data under some
circumstances.
Could you please help me to understand the reason of data corruption?
Please file a bug and attach:
- /var/log/vdsm/vdsm.log
- /var/log/messages
- /var/log/sanlock.log
- output of nfsstat during the test, maybe run it every minute?
vdsm-4.17.13-1.el7.noarch
qemu-img-ev-2.3.0-31.el7_2.4.1.x86_64
libvirt-daemon-1.2.17-13.el7_2.2.x86_64
ovirt-engine-backend-3.6.1.3-1.el7.centos.noarch
Thank you
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:13 p.m.
--_000_1453407210332141camelacroniscom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
T24gVGh1LCAyMDE2LTAxLTIxIGF0IDE4OjQyICswMDAwLCBOaXIgU29mZmVyIHdyb3RlOg0KDQpP
biBUaHUsIEphbiAyMSwgMjAxNiBhdCAyOjU0IFBNLCBQYXZlbCBHYXNoZXYgPFBheEBhY3Jvbmlz
LmNvbTxtYWlsdG86UGF4QGFjcm9uaXMuY29tPj4gd3JvdGU6DQoNCkFsc28gdGhlcmUgaXMgbm8g
b3B0aW9uIGluDQpvVmlydCB3ZWIgaW50ZXJmYWNlIHRvIHVzZSBDT1cgZm9ybWF0IG9uIE5GUyBz
dG9yYWdlIGRvbWFpbnMuDQoNCg0KDQpZb3UgY2FuDQoxLiBjcmVhdGUgYSBzbWFsbCBkaXNrICgx
RykNCjIuIGNyZWF0ZSBhIHNuYXBzaG90DQozLiBleHRlbmQgdGhlIGRpc2sgZ28gdGhlIGZpbmFs
IHNpemUNCg0KQW5kIHlvdSBoYXZlIG5mcyB3aXRoIGNvdyBmb3JtYXQuIFRoZSBwZXJmb3JtYW5j
ZSBkaWZmZXJlbmNlIHdpdGggb25lIHNuYXBzaG90DQpzaG91bGQgYmUgc21hbGwuDQoNCg0KWWVz
LiBBbmQgdGhlcmUgYXJlIG90aGVyIHdvcmthcm91bmRzOg0KMS4gVXNlIHNvbWUgYmxvY2sgKGku
ZS4gaVNDU0kpIHN0b3JhZ2UgZm9yIGNyZWF0aW5nIGEgdGhpbiBwcm92aXNpb25lZCBkaXNrICh3
aGljaCBpcyBDT1cpIGFuZCB0aGVuIG1vdmUgaXQgdG8gcmVxdWlyZWQgc3RvcmFnZS4NCjIuIEtl
ZXAgYW4gZW1wdHkgMUcgQ09XIGRpc2sgYW5kIGNvcHkrcmVzaXplIGl0IHdoZW4gcmVxdWlyZWQu
DQozLiBVc2Ugb3ZpcnQtc2hlbGwgZm9yIGNyZWF0aW5nIGRpc2tzLg0KDQpVbmZvcnR1bmF0ZWx5
LCB0aGVzZSBhcmUgbm90IG5hdGl2ZSB3YXlzLiBUaGVzZSBhcmUgd2F5cyBmb3IgYSBoYWNrZXIu
IFBsYWluIHVzZXIgY2xpY2tzICJOZXciIGluICJEaXNrcyIgdGFiIGFuZCBzZWxlY3RzICJUaGlu
IFByb3Zpc2lvbiIgYWxsb2NhdGlvbiBwb2xpY3kuIEl0J3MgaGFyZCB0byBleHBsYWluIHRvIHVz
ZXJzIHRoYXQgdGhlIHNpbXBsZXN0IGFuZCBvYnZpb3VzIHdheSBpcyB3cm9uZy4gSSBob3BlIGl0
J3Mgd3Jvbmcgb25seSBmb3IgTVMgTkZTLg0KDQoNCjUuIERhdGEgY29ycnVwdGlvbiBoYXBwZW5z
IGFmdGVyICdBdXRvLWdlbmVyYXRlZCBmb3IgTGl2ZSBTdG9yYWdlIE1pZ3JhdGlvbicNCnNuYXBz
aG90LiBTbyBpZiB5b3Ugcm9sbGJhY2sgdGhlIHNuYXBzaG90LCB5b3UgY291bGQgc2VlIGFic29s
dXRlbHkgY2xlYW4NCmZpbGVzeXN0ZW0uDQoNCg0KDQpDYW4geW91IHRyeSB0byBjcmVhdGUgYSBs
aXZlLXNuYXBzaG90IG9uIE1TIE5GUz8gSXQgc2VlbXMgdGhhdCB0aGlzIGlzIHRoZQ0KaXNzdWUs
IG5vdCBsaXZlIHN0b3JhZ2UgbWlncmF0aW9uLg0KDQoNCkxpdmUgc25hcHNob3RzIHdvcmsgdmVy
eSB3ZWxsIG9uIE1TIE5GUy4gQ3JlYXRpbmcgYW5kIGRlbGV0aW5nIHdvcmtzIGxpdmUgd2l0aG91
dCBhbnkgaXNzdWVzLiBJIGRpZCBpdCBtYW55IHRpbWVzLiBQbGVhc2Ugbm90ZSB0aGF0IGV2ZXJ5
dGhpbmcgYmVmb3JlIHRoZSBzbmFwc2hvdCByZW1haW5zIGNvbnNpc3RlbnQuIERhdGEgY29ycnVw
dGlvbiBvY2N1cnMgYWZ0ZXIgdGhlIHNuYXBzaG90LiBTbyBvbmx5IG5vbi1zbmFwc2hvdHRlZCBk
YXRhIGlzIGNvcnJ1cHRlZC4NCg0KDQoNCkRvIHlvdSBoYXZlIHFlbXUtZ3Vlc3QtYWdlbnQgb24g
dGhlIHZtPyBXaXRob3V0IHFlbXUtZ3Vlc3QtYWdlbnQsIGZpbGUNCnN5c3RlbXMgb24gdGhlIGd1
ZXN0IHdpbGwgbm8gYmUgZnJlZXplZCBkdXJpbmcgdGhlIHNuYXBzaG90LCB3aGljaCBtYXkgY2F1
c2UNCmluY29uc2lzdGVudCBzbmFwc2hvdC4NCg0KDQpJIHRyaWVkIGl0IHdpdGggYW5kIHdpdGhv
dXQgcWVtdS1ndWVzdC1hZ2VudC4gSXQgZG9lc24ndCBkZXBlbmQuDQoNCg0KDQpDYW4geW91IHJl
cHJvZHVjZSB0aGlzIHdpdGggdmlydC1tYW5hZ2VyLCBvciBieSBjcmVhdGluZyBhIHZtIGFuZCB0
YWtpbmcNCmEgc25hcHNob3QgdXNpbmcgdmlyc2g/DQoNCg0KU29ycnksIEknbSBub3Qgc3VyZSBo
b3cgSSBjYW4gcmVwcm9kdWNlIHRoZSBpc3N1ZSB1c2luZyB2aXJzaC4NCg0KDQoNCg0KUGxlYXNl
IGZpbGUgYSBidWcgYW5kIGF0dGFjaDoNCg0KLSAvdmFyL2xvZy92ZHNtL3Zkc20ubG9nDQotIC92
YXIvbG9nL21lc3NhZ2VzDQotIC92YXIvbG9nL3NhbmxvY2subG9nDQotIG91dHB1dCBvZiAgbmZz
c3RhdCBkdXJpbmcgdGhlIHRlc3QsIG1heWJlIHJ1biBpdCBldmVyeSBtaW51dGU/DQoNCk9rLCBJ
IHdpbGwgY29sbGVjdCB0aGUgbG9ncyBhbmQgZmlsbCBhIGJ1Zy4NCg0KVGhhbmtzDQoNCg0K
--_000_1453407210332141camelacroniscom_
Content-Type: text/html; charset="utf-8"
Content-ID: <D90DD11001F94349B433588E218DEC19(a)acronis.com>
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5Pg0KPGRpdj5PbiBUaHUs
IDIwMTYtMDEtMjEgYXQgMTg6NDIgJiM0MzswMDAwLCBOaXIgU29mZmVyIHdyb3RlOjwvZGl2Pg0K
PGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+DQo8cHJlPk9uIFRodSwgSmFuIDIxLCAyMDE2IGF0IDI6
NTQgUE0sIFBhdmVsIEdhc2hldiAmbHQ7PGEgaHJlZj0ibWFpbHRvOlBheEBhY3JvbmlzLmNvbSI+
UGF4QGFjcm9uaXMuY29tPC9hPiZndDsgd3JvdGU6DQo8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj5B
bHNvIHRoZXJlIGlzIG5vIG9wdGlvbiBpbg0Kb1ZpcnQgd2ViIGludGVyZmFjZSB0byB1c2UgQ09X
IGZvcm1hdCBvbiBORlMgc3RvcmFnZSBkb21haW5zLg0KPC9ibG9ja3F1b3RlPg0KDQpZb3UgY2Fu
DQoxLiBjcmVhdGUgYSBzbWFsbCBkaXNrICgxRykNCjIuIGNyZWF0ZSBhIHNuYXBzaG90DQozLiBl
eHRlbmQgdGhlIGRpc2sgZ28gdGhlIGZpbmFsIHNpemUNCg0KQW5kIHlvdSBoYXZlIG5mcyB3aXRo
IGNvdyBmb3JtYXQuIFRoZSBwZXJmb3JtYW5jZSBkaWZmZXJlbmNlIHdpdGggb25lIHNuYXBzaG90
DQpzaG91bGQgYmUgc21hbGwuDQo8L3ByZT4NCjwvYmxvY2txdW90ZT4NCjxkaXY+PGJyPg0KPC9k
aXY+DQo8ZGl2Pg0KPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IG1vbm9zcGFjZTsiPlllcy4gQW5k
IHRoZXJlIGFyZSBvdGhlciB3b3JrYXJvdW5kczo8L2Rpdj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFt
aWx5OiBtb25vc3BhY2U7Ij4xLiBVc2Ugc29tZSBibG9jayAoaS5lLiBpU0NTSSkgc3RvcmFnZSBm
b3IgY3JlYXRpbmcgYSB0aGluIHByb3Zpc2lvbmVkIGRpc2sgKHdoaWNoIGlzIENPVykgYW5kIHRo
ZW4gbW92ZSBpdCB0byByZXF1aXJlZCBzdG9yYWdlLjwvZGl2Pg0KPGRpdiBzdHlsZT0iZm9udC1m
YW1pbHk6IG1vbm9zcGFjZTsiPjIuIEtlZXAgYW4gZW1wdHkgMUcgQ09XIGRpc2sgYW5kIGNvcHkm
IzQzO3Jlc2l6ZSBpdCB3aGVuIHJlcXVpcmVkLjwvZGl2Pg0KPGRpdiBzdHlsZT0iZm9udC1mYW1p
bHk6IG1vbm9zcGFjZTsiPjMuIFVzZSBvdmlydC1zaGVsbCBmb3IgY3JlYXRpbmcgZGlza3MuPC9k
aXY+DQo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTogbW9ub3NwYWNlOyI+PGJyPg0KPC9kaXY+DQo8
ZGl2IHN0eWxlPSJmb250LWZhbWlseTogbW9ub3NwYWNlOyI+VW5mb3J0dW5hdGVseSwgdGhlc2Ug
YXJlIG5vdCBuYXRpdmUgd2F5cy4gVGhlc2UgYXJlIHdheXMgZm9yIGEgaGFja2VyLiBQbGFpbiB1
c2VyIGNsaWNrcyAmcXVvdDtOZXcmcXVvdDsgaW4gJnF1b3Q7RGlza3MmcXVvdDsgdGFiIGFuZCBz
ZWxlY3RzICZxdW90O1RoaW4gUHJvdmlzaW9uJnF1b3Q7IGFsbG9jYXRpb24gcG9saWN5LiBJdCdz
IGhhcmQgdG8gZXhwbGFpbiB0byB1c2VycyB0aGF0IHRoZSBzaW1wbGVzdCBhbmQgb2J2aW91cyB3
YXkNCiBpcyB3cm9uZy4gSSBob3BlIGl0J3Mgd3Jvbmcgb25seSBmb3IgTVMgTkZTLjwvZGl2Pg0K
PC9kaXY+DQo8ZGl2Pjxicj4NCjwvZGl2Pg0KPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+DQo8cHJl
PjxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPjUuIERhdGEgY29ycnVwdGlvbiBoYXBwZW5zIGFmdGVy
ICdBdXRvLWdlbmVyYXRlZCBmb3IgTGl2ZSBTdG9yYWdlIE1pZ3JhdGlvbicNCnNuYXBzaG90LiBT
byBpZiB5b3Ugcm9sbGJhY2sgdGhlIHNuYXBzaG90LCB5b3UgY291bGQgc2VlIGFic29sdXRlbHkg
Y2xlYW4NCmZpbGVzeXN0ZW0uDQo8L2Jsb2NrcXVvdGU+DQoNCkNhbiB5b3UgdHJ5IHRvIGNyZWF0
ZSBhIGxpdmUtc25hcHNob3Qgb24gTVMgTkZTPyBJdCBzZWVtcyB0aGF0IHRoaXMgaXMgdGhlDQpp
c3N1ZSwgbm90IGxpdmUgc3RvcmFnZSBtaWdyYXRpb24uDQo8L3ByZT4NCjwvYmxvY2txdW90ZT4N
CjxkaXY+PGJyPg0KPC9kaXY+DQo8ZGl2PkxpdmUgc25hcHNob3RzIHdvcmsgdmVyeSB3ZWxsIG9u
IE1TIE5GUy4gQ3JlYXRpbmcgYW5kIGRlbGV0aW5nIHdvcmtzIGxpdmUgd2l0aG91dCBhbnkgaXNz
dWVzLiBJIGRpZCBpdCBtYW55IHRpbWVzLiBQbGVhc2Ugbm90ZSB0aGF0IGV2ZXJ5dGhpbmcgYmVm
b3JlIHRoZSBzbmFwc2hvdCByZW1haW5zIGNvbnNpc3RlbnQuIERhdGEgY29ycnVwdGlvbiBvY2N1
cnMgYWZ0ZXIgdGhlIHNuYXBzaG90LiBTbyBvbmx5IG5vbi1zbmFwc2hvdHRlZCBkYXRhDQogaXMg
Y29ycnVwdGVkLjwvZGl2Pg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxibG9ja3F1b3RlIHR5cGU9ImNp
dGUiPg0KPHByZT4NCkRvIHlvdSBoYXZlIHFlbXUtZ3Vlc3QtYWdlbnQgb24gdGhlIHZtPyBXaXRo
b3V0IHFlbXUtZ3Vlc3QtYWdlbnQsIGZpbGUNCnN5c3RlbXMgb24gdGhlIGd1ZXN0IHdpbGwgbm8g
YmUgZnJlZXplZCBkdXJpbmcgdGhlIHNuYXBzaG90LCB3aGljaCBtYXkgY2F1c2UNCmluY29uc2lz
dGVudCBzbmFwc2hvdC4NCjwvcHJlPg0KPC9ibG9ja3F1b3RlPg0KPGRpdj48YnI+DQo8L2Rpdj4N
CjxkaXY+SSB0cmllZCBpdCB3aXRoIGFuZCB3aXRob3V0IHFlbXUtZ3Vlc3QtYWdlbnQuIEl0IGRv
ZXNuJ3QgZGVwZW5kLiZuYnNwOzwvZGl2Pg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxibG9ja3F1b3Rl
IHR5cGU9ImNpdGUiPg0KPHByZT4NCkNhbiB5b3UgcmVwcm9kdWNlIHRoaXMgd2l0aCB2aXJ0LW1h
bmFnZXIsIG9yIGJ5IGNyZWF0aW5nIGEgdm0gYW5kIHRha2luZw0KYSBzbmFwc2hvdCB1c2luZyB2
aXJzaD8NCjwvcHJlPg0KPC9ibG9ja3F1b3RlPg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+U29y
cnksIEknbSBub3Qgc3VyZSBob3cgSSBjYW4gcmVwcm9kdWNlIHRoZSBpc3N1ZSB1c2luZyB2aXJz
aC48L2Rpdj4NCjxkaXY+PGJyPg0KPC9kaXY+DQo8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4NCjxw
cmU+DQoNClBsZWFzZSBmaWxlIGEgYnVnIGFuZCBhdHRhY2g6DQoNCi0gL3Zhci9sb2cvdmRzbS92
ZHNtLmxvZw0KLSAvdmFyL2xvZy9tZXNzYWdlcw0KLSAvdmFyL2xvZy9zYW5sb2NrLmxvZw0KLSBv
dXRwdXQgb2YgIG5mc3N0YXQgZHVyaW5nIHRoZSB0ZXN0LCBtYXliZSBydW4gaXQgZXZlcnkgbWlu
dXRlPzwvcHJlPg0KPC9ibG9ja3F1b3RlPg0KPGRpdj48YnI+DQo8L2Rpdj4NCjxkaXY+T2ssIEkg
d2lsbCBjb2xsZWN0IHRoZSBsb2dzIGFuZCBmaWxsIGEgYnVnLjwvZGl2Pg0KPGRpdj48YnI+DQo8
L2Rpdj4NCjxkaXY+VGhhbmtzPC9kaXY+DQo8ZGl2Pjxicj4NCjwvZGl2Pg0KPGRpdj48YnI+DQo8
L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg==
--_000_1453407210332141camelacroniscom_--
2:55 p.m.
On Thu, Jan 21, 2016 at 10:13 PM, Pavel Gashev <Pax(a)acronis.com> wrote:
On Thu, 2016-01-21 at 18:42 +0000, Nir Soffer wrote:
On Thu, Jan 21, 2016 at 2:54 PM, Pavel Gashev <Pax(a)acronis.com> wrote:
Also there is no option in
oVirt web interface to use COW format on NFS storage domains.
You can
1. create a small disk (1G)
2. create a snapshot
3. extend the disk go the final size
And you have nfs with cow format. The performance difference with one
snapshot
should be small.
Yes. And there are other workarounds:
1. Use some block (i.e. iSCSI) storage for creating a thin provisioned disk
(which is COW) and then move it to required storage.
2. Keep an empty 1G COW disk and copy+resize it when required.
3. Use ovirt-shell for creating disks.
Unfortunately, these are not native ways. These are ways for a hacker. Plain
user clicks "New" in "Disks" tab and selects "Thin
Provision" allocation
policy. It's hard to explain to users that the simplest and obvious way is
wrong. I hope it's wrong only for MS NFS.
Sure I agree.
I think we do not use qcow format on file storage since there is no
need for this,
the file system is always sparse. I guess we did not plan to use MS NFS.
I would open bug for supporting qcow format on file storage. If this works for
some users, I think this is an option that should be possible in the
ui. Hopefully
there are no too many assumptions in the code about this.
Allon, do you see any reason not to support this for user that need this option?
5. Data corruption happens after 'Auto-generated for Live Storage Migration'
snapshot. So if you rollback the snapshot, you could see absolutely clean
filesystem.
Can you try to create a live-snapshot on MS NFS? It seems that this is the
issue, not live storage migration.
Live snapshots work very well on MS NFS. Creating and deleting works live
without any issues. I did it many times. Please note that everything before
the snapshot remains consistent. Data corruption occurs after the snapshot.
So only non-snapshotted data is corrupted.
live migration starts by creating a snapshot, then copying the disks to the new
storage, and then mirroring the active layer so both the old and the
new disks are
the same. Finally we switch to the new disk, and delete the old disk.
So probably the issue is in the mirroring step. This is most likely a
qemu issue.
Do you have qemu-guest-agent on the vm? Without qemu-guest-agent, file
systems on the guest will no be freezed during the snapshot, which may cause
inconsistent snapshot.
I tried it with and without qemu-guest-agent. It doesn't depend.
Can you reproduce this with virt-manager, or by creating a vm and taking
a snapshot using virsh?
Sorry, I'm not sure how I can reproduce the issue using virsh.
I'll try to get instructions for this from libvirt developers. If this
happen with
libvirt alone, this is a libvirt or qemu bug, and there is little we (ovirt) can
do about it.
Please file a bug and attach:
- /var/log/vdsm/vdsm.log
- /var/log/messages
- /var/log/sanlock.log
- output of nfsstat during the test, maybe run it every minute?
Ok, I will collect the logs and fill a bug.
Thanks
5:23 p.m.
Adding Allon
On Thu, Jan 21, 2016 at 10:55 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Thu, Jan 21, 2016 at 10:13 PM, Pavel Gashev
<Pax(a)acronis.com> wrote:
> On Thu, 2016-01-21 at 18:42 +0000, Nir Soffer wrote:
>
> On Thu, Jan 21, 2016 at 2:54 PM, Pavel Gashev <Pax(a)acronis.com> wrote:
>
> Also there is no option in
> oVirt web interface to use COW format on NFS storage domains.
>
>
> You can
> 1. create a small disk (1G)
> 2. create a snapshot
> 3. extend the disk go the final size
>
> And you have nfs with cow format. The performance difference with one
> snapshot
> should be small.
>
>
> Yes. And there are other workarounds:
> 1. Use some block (i.e. iSCSI) storage for creating a thin provisioned disk
> (which is COW) and then move it to required storage.
> 2. Keep an empty 1G COW disk and copy+resize it when required.
> 3. Use ovirt-shell for creating disks.
>
> Unfortunately, these are not native ways. These are ways for a hacker. Plain
> user clicks "New" in "Disks" tab and selects "Thin
Provision" allocation
> policy. It's hard to explain to users that the simplest and obvious way is
> wrong. I hope it's wrong only for MS NFS.
Sure I agree.
I think we do not use qcow format on file storage since there is no
need for this,
the file system is always sparse. I guess we did not plan to use MS NFS.
I would open bug for supporting qcow format on file storage. If this works for
some users, I think this is an option that should be possible in the
ui. Hopefully
there are no too many assumptions in the code about this.
Allon, do you see any reason not to support this for user that need this option?
>
> 5. Data corruption happens after 'Auto-generated for Live Storage Migration'
> snapshot. So if you rollback the snapshot, you could see absolutely clean
> filesystem.
>
>
> Can you try to create a live-snapshot on MS NFS? It seems that this is the
> issue, not live storage migration.
>
>
> Live snapshots work very well on MS NFS. Creating and deleting works live
> without any issues. I did it many times. Please note that everything before
> the snapshot remains consistent. Data corruption occurs after the snapshot.
> So only non-snapshotted data is corrupted.
live migration starts by creating a snapshot, then copying the disks to the new
storage, and then mirroring the active layer so both the old and the
new disks are
the same. Finally we switch to the new disk, and delete the old disk.
So probably the issue is in the mirroring step. This is most likely a
qemu issue.
>
> Do you have qemu-guest-agent on the vm? Without qemu-guest-agent, file
> systems on the guest will no be freezed during the snapshot, which may cause
> inconsistent snapshot.
>
>
> I tried it with and without qemu-guest-agent. It doesn't depend.
>
> Can you reproduce this with virt-manager, or by creating a vm and taking
> a snapshot using virsh?
>
>
> Sorry, I'm not sure how I can reproduce the issue using virsh.
I'll try to get instructions for this from libvirt developers. If this
happen with
libvirt alone, this is a libvirt or qemu bug, and there is little we (ovirt) can
do about it.
>
>
> Please file a bug and attach:
>
> - /var/log/vdsm/vdsm.log
> - /var/log/messages
> - /var/log/sanlock.log
> - output of nfsstat during the test, maybe run it every minute?
>
>
> Ok, I will collect the logs and fill a bug.
>
> Thanks
>
>
9:15 a.m.
Nir,
On 21/01/16 23:55, "Nir Soffer" <nsoffer(a)redhat.com> wrote:
live migration starts by creating a snapshot, then copying the disks
to the new
storage, and then mirroring the active layer so both the old and the
new disks are
the same. Finally we switch to the new disk, and delete the old disk.
So probably the issue is in the mirroring step. This is most likely a
qemu issue.
Thank you for clarification. This brought me an idea to check consistency of the old
disk.
I performed the following testing:
1. Create a VM on MS NFS
2. Initiate live disk migration to another storage
3. Catch the source files before oVirt has removed them by creating hard links to another
directory
4. Shutdown VM
5. Create another VM and move the catched files to the place where new disk files is
located
6. Check consistency of filesystem in both VMs
The source disk is consistent. The destination disk is corrupted.
I'll try to get instructions for this from libvirt developers. If this
happen with
libvirt alone, this is a libvirt or qemu bug, and there is little we (ovirt) can
do about it.
I've tried to reproduce the mirroring of active layer:
1. Create two thin template provisioned VMs from the same template on different
storages.
2. Start VM1
3. virsh blockcopy VM1 vda /rhev/data-center/...path.to.disk.of.VM2.. --wait --verbose
--reuse-external --shallow
4. virsh blockjob VM1 vda --abort --pivot
5. Shutdown VM1
6. Start VM2. Boot in recovery mode and check filesystem.
I did try this a dozen times. Everything works fine. No data corruption.
Ideas?
2:47 p.m.
On Fri, Jan 22, 2016 at 5:15 PM, Pavel Gashev <Pax(a)acronis.com> wrote:
Nir,
On 21/01/16 23:55, "Nir Soffer" <nsoffer(a)redhat.com> wrote:
>live migration starts by creating a snapshot, then copying the disks to the new
>storage, and then mirroring the active layer so both the old and the
>new disks are
>the same. Finally we switch to the new disk, and delete the old disk.
>
>So probably the issue is in the mirroring step. This is most likely a
>qemu issue.
Thank you for clarification. This brought me an idea to check consistency of the old
disk.
I performed the following testing:
1. Create a VM on MS NFS
2. Initiate live disk migration to another storage
3. Catch the source files before oVirt has removed them by creating hard links to another
directory
4. Shutdown VM
5. Create another VM and move the catched files to the place where new disk files is
located
6. Check consistency of filesystem in both VMs
The source disk is consistent. The destination disk is corrupted.
>
>I'll try to get instructions for this from libvirt developers. If this
>happen with
>libvirt alone, this is a libvirt or qemu bug, and there is little we (ovirt) can
>do about it.
I've tried to reproduce the mirroring of active layer:
1. Create two thin template provisioned VMs from the same template on different
storages.
2. Start VM1
3. virsh blockcopy VM1 vda /rhev/data-center/...path.to.disk.of.VM2.. --wait --verbose
--reuse-external --shallow
4. virsh blockjob VM1 vda --abort --pivot
5. Shutdown VM1
6. Start VM2. Boot in recovery mode and check filesystem.
I did try this a dozen times. Everything works fine. No data corruption.
If you take same vm, and do a live storage migration in ovirt, the
file system is
corrupted after the migration?
What is the guest os? did you try with more then one?
Ideas?
Thanks for this research!
The next step is to open a bug with the logs I requested in my last
message. Please mark the bug
as urgent.
I'm adding Kevin (from qemu) and Eric (from libvirt), hopefully they
can tell if the virsh flow is
indeed identical to what ovirt does, and what should be the next step
for debugging this.
Ovirt is using blockCopy if available (should be available everywhere
for some time), or fallback
to blockRebase. Do you see this warning?
blockCopy not supported, using blockRebase
For reference, this is the relevant code in ovirt for the mirroring
part. The mirroring starts with
diskReplicateStart(), and ends with diskReplicateFinish(). I remove
the parts about managing
vdsm state and left the calls to libvirt.
3378 def diskReplicateFinish(self, srcDisk, dstDisk):
...
3394 blkJobInfo = self._dom.blockJobInfo(drive.name, 0)
...
3418 if srcDisk != dstDisk:
3419 self.log.debug("Stopping the disk replication
switching to the "
3420 "destination drive: %s", dstDisk)
3421 blockJobFlags = libvirt.VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT
...
3429 else:
3430 self.log.debug("Stopping the disk replication
remaining on the "
3431 "source drive: %s", dstDisk)
3432 blockJobFlags = 0
...
3435 try:
3436 # Stopping the replication
3437 self._dom.blockJobAbort(drive.name, blockJobFlags)
3438 except Exception:
3439 self.log.exception("Unable to stop the replication for"
3440 " the drive: %s", drive.name)
...
3462 def _startDriveReplication(self, drive):
3463 destxml = drive.getReplicaXML().toprettyxml()
3464 self.log.debug("Replicating drive %s to %s", drive.name, destxml)
3465
3466 flags = (libvirt.VIR_DOMAIN_BLOCK_COPY_SHALLOW |
3467 libvirt.VIR_DOMAIN_BLOCK_COPY_REUSE_EXT)
3468
3469 # TODO: Remove fallback when using libvirt >= 1.2.9.
3470 try:
3471 self._dom.blockCopy(drive.name, destxml, flags=flags)
3472 except libvirt.libvirtError as e:
3473 if e.get_error_code() != libvirt.VIR_ERR_NO_SUPPORT:
3474 raise
3475
3476 self.log.warning("blockCopy not supported, using blockRebase")
3477
3478 base = drive.diskReplicate["path"]
3479 self.log.debug("Replicating drive %s to %s", drive.name, base)
3480
3481 flags = (libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY |
3482 libvirt.VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT |
3483 libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW)
3484
3485 if drive.diskReplicate["diskType"] == DISK_TYPE.BLOCK:
3486 flags |= libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY_DEV
3487
3488 self._dom.blockRebase(drive.name, base, flags=flags)
1:20 p.m.
Nir,
On Fri, 2016-01-22 at 20:47 +0000, Nir Soffer wrote:
On Fri, Jan 22, 2016 at 5:15 PM, Pavel Gashev
<Pax(a)acronis.com>
wrote:
> I've tried to reproduce the mirroring of active layer:
>
> 1. Create two thin template provisioned VMs from the same template
> on different storages.
> 2. Start VM1
> 3. virsh blockcopy VM1 vda /rhev/data
> -center/...path.to.disk.of.VM2.. --wait --verbose --reuse-external
> --shallow
> 4. virsh blockjob VM1 vda --abort --pivot
> 5. Shutdown VM1
> 6. Start VM2. Boot in recovery mode and check filesystem.
>
> I did try this a dozen times. Everything works fine. No data
> corruption.
If you take same vm, and do a live storage migration in ovirt, the
file system is
corrupted after the migration?
Yes. And I've reproduced the issue:
1. Create a VM on MS NFS
2. Start VM
3. Create a disk-only snapshot
4. virsh blockcopy VM1 /some/file --wait --verbose --reuse-external
--shallow
5. virsh blockjob VM1 vda --abort --pivot
6. Shutdown VM
7. Copy the /some/file back to
/rhev/data-center/..the.latest.snapshot.of.VM..
8. Start VM and check filesystem
In other words, creating a snapshot is important step for reproducing
the issue.
What is the guest os? did you try with more then one?
Guest OS is W2K12. I was unable to reproduce the issue with Linux.
The next step is to open a bug with the logs I requested in my last
message. Please mark the bug
as urgent.
https://bugzilla.redhat.com/show_bug.cgi?id=1301713
I'm adding Kevin (from qemu) and Eric (from libvirt), hopefully
they
can tell if the virsh flow is
indeed identical to what ovirt does, and what should be the next step
for debugging this.
Ovirt is using blockCopy if available (should be available everywhere
for some time), or fallback
to blockRebase. Do you see this warning?
blockCopy not supported, using blockRebase
No such warning. There is 'Replicating drive vda to <disk...'
Please find vdsm.log attached to the bug report.
Thanks
5:45 a.m.
On Mon, Jan 25, 2016 at 9:20 PM, Pavel Gashev <Pax(a)acronis.com> wrote:
Nir,
On Fri, 2016-01-22 at 20:47 +0000, Nir Soffer wrote:
> On Fri, Jan 22, 2016 at 5:15 PM, Pavel Gashev <Pax(a)acronis.com>
> wrote:
> > I've tried to reproduce the mirroring of active layer:
> >
> > 1. Create two thin template provisioned VMs from the same template
> > on different storages.
> > 2. Start VM1
> > 3. virsh blockcopy VM1 vda /rhev/data
> > -center/...path.to.disk.of.VM2.. --wait --verbose --reuse-external
> > --shallow
> > 4. virsh blockjob VM1 vda --abort --pivot
> > 5. Shutdown VM1
> > 6. Start VM2. Boot in recovery mode and check filesystem.
> >
> > I did try this a dozen times. Everything works fine. No data
> > corruption.
>
> If you take same vm, and do a live storage migration in ovirt, the
> file system is
> corrupted after the migration?
Yes. And I've reproduced the issue:
1. Create a VM on MS NFS
2. Start VM
3. Create a disk-only snapshot
4. virsh blockcopy VM1 /some/file --wait --verbose --reuse-external
--shallow
5. virsh blockjob VM1 vda --abort --pivot
At this point, /some/file should be the top layer of the vm, instead of
the latest snapshot of the vm.
Can you add the output of qemu-img info /some/file?
6. Shutdown VM
7. Copy the /some/file back to
/rhev/data-center/..the.latest.snapshot.of.VM..
Can you add the output of qemu-img info on the lastest snapshot of the vm?
8. Start VM and check filesystem
What are the results of this check?
Can you answer this on the bug, so we have more information for libvirt and qemu
developers?
In other words, creating a snapshot is important step for reproducing
the issue.
So this happens only when mirroring a volume that is a result of a live
snapshot, right?
> What is the guest os? did you try with more then one?
Guest OS is W2K12. I was unable to reproduce the issue with Linux.
W2K - Windows 2000?
> The next step is to open a bug with the logs I requested in my last
> message. Please mark the bug
> as urgent.
https://bugzilla.redhat.com/show_bug.cgi?id=1301713
> I'm adding Kevin (from qemu) and Eric (from libvirt), hopefully they
> can tell if the virsh flow is
> indeed identical to what ovirt does, and what should be the next step
> for debugging this.
>
> Ovirt is using blockCopy if available (should be available everywhere
> for some time), or fallback
> to blockRebase. Do you see this warning?
>
> blockCopy not supported, using blockRebase
No such warning. There is 'Replicating drive vda to <disk...'
Please find vdsm.log attached to the bug report.
Thanks
3226
days inactive
3231
days old
9 comments
3 participants
participants (3)
-
Dan Yasny -
Nir Soffer -
Pavel Gashev