[ovirt-users] storage issue's with oVirt 3.5.1 + Nexenta NFS

[Maikel vd Mosselaar]

Hi Fred,


This is one of the nodes from yesterday around 01:00 (20-04-15). The 
issue started around 01:00.
https://bpaste.net/raw/67542540a106

The VDSM logs are very big so i am unable to paste a bigger part of the 
logfile, i don't know what the maximum allowed attachment size is of the 
mailing list?

dmesg on the one the nodes (despite this message the storage is still 
accessible):
https://bpaste.net/raw/67da167aa300



Kind regards,

Maikel

On 04/21/2015 02:32 PM, Fred Rolland wrote:
> Hi,
>
> Can you please attach VDSM logs ?
>
> Thanks,
>
> Fred
>
> ----- Original Message -----
>> From: "Maikel vd Mosselaar" <m.vandemosselaar at smoose.nl>
>> To: users at ovirt.org
>> Sent: Monday, April 20, 2015 3:25:38 PM
>> Subject: [ovirt-users] storage issue's with oVirt 3.5.1 + Nexenta NFS
>>
>> Hi,
>>
>> We are running ovirt 3.5.1 with 3 nodes and seperate engine.
>>
>> All on CentOS 6.6:
>> 3 x nodes
>> 1 x engine
>>
>> 1 x storage nexenta with NFS
>>
>> For multiple weeks we are experiencing issues of our nodes that cannot
>> access the storage at random moments (atleast thats what the nodes think).
>>
>> When the nodes are complaining about a unavailable storage then the load
>> rises up to +200 on all three nodes, this causes that all running VMs
>> are unaccessible. During this process oVirt event viewer shows some i/o
>> storage error messages, when this happens random VMs get paused and will
>> not be resumed anymore (this almost happens every time but not all the
>> VMs get paused).
>>
>> During the event we tested the accessibility from the nodes to the
>> storage and it looks like it is working normal, at least we can do a normal
>> "ls" on the storage without any delay of showing the contents.
>>
>> We tried multiple things that we thought it causes this issue but
>> nothing worked so far.
>> * rebooting storage / nodes / engine.
>> * disabling offsite rsync backups.
>> * moved the biggest VMs with highest load to different platform outside
>> of oVirt.
>> * checked the wsize and rsize on the nfs mounts, storage and nodes are
>> correct according to the "NFS troubleshooting page" on ovirt.org.
>>
>> The environment is running in production so we are not free to test
>> everything.
>>
>> I can provide log files if needed.
>>
>> Kind Regards,
>>
>> Maikel
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users