Hi guys,

I had a strange issue on the 3rd of September and I've only got round to checking what caused it now. Basically about 4 or 5 Windows Server VM's got completely corrupted. When I press start I'd just get a blank screen and nothing would display, I tried various things but no matter what I wouldn't even get the Seabios display showing the VM was even posting
The remaining 10 VM's were fine, it was just these 4 or 5 that got corrupted and to recover I had to do a full DR restore of the VM's.

I'm concerned that the issue might appear again, which is why I'm mailing the list now, does anyone have any clues as to what might have caused this? All logs on the FC SAN were fine and all hosts appeared normal...

The following are my versions...

CentOS release 6.5 (Final)
ovirt-release34-1.0.3-1.noarch
ovirt-host-deploy-1.2.3-1.el6.noarch
ovirt-engine-lib-3.4.4-1.el6.noarch
ovirt-iso-uploader-3.4.4-1.el6.noarch
ovirt-engine-cli-3.4.0.5-1.el6.noarch
ovirt-engine-setup-base-3.4.4-1.el6.noarch
ovirt-engine-websocket-proxy-3.4.4-1.el6.noarch
ovirt-engine-backend-3.4.4-1.el6.noarch
ovirt-engine-tools-3.4.4-1.el6.noarch
ovirt-engine-dbscripts-3.4.4-1.el6.noarch
ovirt-engine-3.4.4-1.el6.noarch
ovirt-engine-setup-3.4.4-1.el6.noarch
ovirt-engine-sdk-python-3.4.4.0-1.el6.noarch
ovirt-image-uploader-3.4.3-1.el6.noarch
ovirt-host-deploy-java-1.2.3-1.el6.noarch
ovirt-engine-setup-plugin-websocket-proxy-3.4.4-1.el6.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-3.4.4-1.el6.noarch
ovirt-engine-restapi-3.4.4-1.el6.noarch
ovirt-engine-userportal-3.4.4-1.el6.noarch
ovirt-engine-webadmin-portal-3.4.4-1.el6.noarch
ovirt-engine-setup-plugin-ovirt-engine-3.4.4-1.el6.noarch

CentOS release 6.5 (Final)
vdsm-python-zombiereaper-4.14.11.2-0.el6.noarch
vdsm-cli-4.14.11.2-0.el6.noarch
vdsm-python-4.14.11.2-0.el6.x86_64
vdsm-4.14.11.2-0.el6.x86_64
vdsm-xmlrpc-4.14.11.2-0.el6.noarch

Below are the sanlock.logs from two of my hosts and attached is my ovirt-engine.log from the date of the issue...

Node02
2015-09-03 10:34:53+0200 33184492 [7369]: 0e6991ae aio timeout 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd9094b000 ioto 10 to_count 7
2015-09-03 10:34:53+0200 33184492 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 10:34:53+0200 33184492 [7369]: s1 renewal error -202 delta_length 10 last_success 33184461
2015-09-03 10:35:04+0200 33184503 [7369]: 0e6991ae aio timeout 0 0x7fbd70000910:0x7fbd70000920:0x7fbd7feff000 ioto 10 to_count 8
2015-09-03 10:35:04+0200 33184503 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 10:35:04+0200 33184503 [7369]: s1 renewal error -202 delta_length 11 last_success 33184461
2015-09-03 10:35:05+0200 33184504 [7369]: 0e6991ae aio collect 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd9094b000 result 1048576:0 other free r
2015-09-03 10:35:05+0200 33184504 [7369]: 0e6991ae aio collect 0 0x7fbd70000910:0x7fbd70000920:0x7fbd7feff000 result 1048576:0 match reap
2015-09-03 11:03:00+0200 33186178 [7369]: 0e6991ae aio timeout 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd7feff000 ioto 10 to_count 9
2015-09-03 11:03:00+0200 33186178 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 11:03:00+0200 33186178 [7369]: s1 renewal error -202 delta_length 10 last_success 33186147
2015-09-03 11:03:07+0200 33186185 [7369]: 0e6991ae aio collect 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd7feff000 result 1048576:0 other free
2015-09-03 11:10:18+0200 33186616 [7369]: 0e6991ae aio timeout 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd9094b000 ioto 10 to_count 10
2015-09-03 11:10:18+0200 33186616 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 11:10:18+0200 33186616 [7369]: s1 renewal error -202 delta_length 10 last_success 33186586
2015-09-03 11:10:21+0200 33186620 [7369]: 0e6991ae aio collect 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd9094b000 result 1048576:0 other free
2015-09-03 12:39:14+0200 33191953 [7369]: 0e6991ae aio timeout 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd7feff000 ioto 10 to_count 11
2015-09-03 12:39:14+0200 33191953 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 12:39:14+0200 33191953 [7369]: s1 renewal error -202 delta_length 10 last_success 33191922
2015-09-03 12:39:19+0200 33191957 [7369]: 0e6991ae aio collect 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd7feff000 result 1048576:0 other free
2015-09-03 12:40:10+0200 33192008 [7369]: 0e6991ae aio timeout 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd9094b000 ioto 10 to_count 12
2015-09-03 12:40:10+0200 33192008 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 12:40:10+0200 33192008 [7369]: s1 renewal error -202 delta_length 10 last_success 33191977
2015-09-03 12:40:12+0200 33192011 [7369]: 0e6991ae aio collect 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd9094b000 result 1048576:0 other free
2015-09-03 12:43:17+0200 33192196 [7369]: 0e6991ae aio timeout 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd7feff000 ioto 10 to_count 13
2015-09-03 12:43:17+0200 33192196 [7369]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 12:43:17+0200 33192196 [7369]: s1 renewal error -202 delta_length 10 last_success 33192165
2015-09-03 12:43:25+0200 33192203 [7369]: 0e6991ae aio collect 0 0x7fbd700008c0:0x7fbd700008d0:0x7fbd7feff000 result 1048576:0 other free
2015-09-03 13:02:43+0200 33193361 [5807]: cmd 9 target pid 23383 not found
2015-09-03 13:13:24+0200 33194002 [5807]: cmd 9 target pid 24611 not found
2015-09-03 13:35:10+0200 33195308 [5807]: cmd 9 target pid 26392 not found
2015-09-03 13:53:32+0200 33196411 [5807]: cmd 9 target pid 28213 not found
2015-09-03 14:33:42+0200 33198820 [5807]: cmd 9 target pid 30732 not found


Node3
2015-09-03 10:34:53+0200 33181297 [7509]: 0e6991ae aio timeout 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45ec434000 ioto 10 to_count 7
2015-09-03 10:34:53+0200 33181297 [7509]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 10:34:53+0200 33181297 [7509]: s1 renewal error -202 delta_length 10 last_success 33181266
2015-09-03 10:35:04+0200 33181308 [7509]: 0e6991ae aio timeout 0 0x7f45d0000910:0x7f45d0000920:0x7f45f03c9000 ioto 10 to_count 8
2015-09-03 10:35:04+0200 33181308 [7509]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 10:35:04+0200 33181308 [7509]: s1 renewal error -202 delta_length 11 last_success 33181266
2015-09-03 10:35:05+0200 33181309 [7509]: 0e6991ae aio collect 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45ec434000 result 1048576:0 other free r
2015-09-03 10:35:05+0200 33181309 [7509]: 0e6991ae aio collect 0 0x7f45d0000910:0x7f45d0000920:0x7f45f03c9000 result 1048576:0 match reap
2015-09-03 11:03:00+0200 33182983 [7509]: 0e6991ae aio timeout 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45f03c9000 ioto 10 to_count 9
2015-09-03 11:03:00+0200 33182983 [7509]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 11:03:00+0200 33182983 [7509]: s1 renewal error -202 delta_length 10 last_success 33182953
2015-09-03 11:03:07+0200 33182990 [7509]: 0e6991ae aio collect 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45f03c9000 result 1048576:0 other free
2015-09-03 11:10:29+0200 33183432 [7509]: s1 renewed 33183417 delta_length 21 too long
2015-09-03 12:31:46+0200 33188310 [5666]: cmd 9 target pid 3657 not found
2015-09-03 12:39:14+0200 33188758 [7509]: 0e6991ae aio timeout 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45ec434000 ioto 10 to_count 10
2015-09-03 12:39:14+0200 33188758 [7509]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 12:39:14+0200 33188758 [7509]: s1 renewal error -202 delta_length 10 last_success 33188727
2015-09-03 12:39:19+0200 33188762 [7509]: 0e6991ae aio collect 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45ec434000 result 1048576:0 other free
2015-09-03 12:40:10+0200 33188813 [7509]: 0e6991ae aio timeout 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45f03c9000 ioto 10 to_count 11
2015-09-03 12:40:10+0200 33188813 [7509]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 12:40:10+0200 33188813 [7509]: s1 renewal error -202 delta_length 10 last_success 33188783
2015-09-03 12:40:12+0200 33188816 [7509]: 0e6991ae aio collect 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45f03c9000 result 1048576:0 other free
2015-09-03 12:43:17+0200 33189001 [7509]: 0e6991ae aio timeout 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45ec434000 ioto 10 to_count 12
2015-09-03 12:43:17+0200 33189001 [7509]: s1 delta_renew read rv -202 offset 0 /dev/0e6991ae-6238-4c61-96d2-ca8fed35161e/ids
2015-09-03 12:43:17+0200 33189001 [7509]: s1 renewal error -202 delta_length 10 last_success 33188970
2015-09-03 12:43:25+0200 33189008 [7509]: 0e6991ae aio collect 0 0x7f45d00008c0:0x7f45d00008d0:0x7f45ec434000 result 1048576:0 other free
2015-09-03 12:54:43+0200 33189687 [5666]: cmd 9 target pid 6503 not found
2015-09-03 13:00:01+0200 33190004 [5666]: cmd 9 target pid 7021 not found
2015-09-03 13:01:20+0200 33190083 [5666]: cmd 9 target pid 8009 not found
2015-09-03 13:06:38+0200 33190401 [5666]: cmd 9 target pid 9119 not found
2015-09-03 13:12:31+0200 33190754 [5666]: cmd 9 target pid 10248 not found
2015-09-03 14:03:36+0200 33193819 [5666]: cmd 9 target pid 13381 not found
2015-09-03 14:05:56+0200 33193959 [5666]: cmd 9 target pid 14367 not found
2015-09-03 14:16:02+0200 33194565 [5666]: cmd 9 target pid 15553 not found
2015-09-03 14:17:01+0200 33194624 [5666]: cmd 9 target pid 16385 not found
2015-09-03 14:23:19+0200 33195002 [5666]: cmd 9 target pid 17456 not found
2015-09-03 14:47:25+0200 33196448 [5666]: cmd 9 target pid 20262 not found
2015-09-03 15:02:45+0200 33197368 [5666]: cmd 9 target pid 21619 not found
2015-09-03 15:03:37+0200 33197420 [5666]: cmd 9 target pid 22321 not found
2015-09-03 15:07:43+0200 33197666 [5666]: cmd 9 target pid 23381 not found
2015-09-03 16:33:39+0200 33202822 [5666]: cmd 9 target pid 29063 not found
2015-09-09 11:36:22+0200 33703385 [5666]: cmd 9 target pid 22695 not found
2015-09-09 11:51:15+0200 33704278 [5666]: cmd 9 target pid 24089 not found
2015-09-09 11:58:25+0200 33704709 [5666]: cmd 9 target pid 25110 not found
2015-09-21 09:29:36+0200 34732579 [5666]: cmd 9 target pid 8527 not found

Please shout if you need more info, unfortunately because I've left this for so long the logs might have rotated already.

Any assistance is greatly appreciated.

Regards.

Neil Wilson.