Hi Thomas,

I am by no means expert, but here couple of thoughts.

The cache filling up could be raid controller cache filling up and not emptying fast enough.
To have valid results, you could use another tool than dd, or run dd multiple times after another.

I don’t think, network have anything to do here with disk load, it would rather create system load as writes get queued.

Raid 4 seems strange to run VMs. Any reason why you pick this? The way I understand is that although you have 3 disks for data, all parity bits from the 3 disks write to the 4th disk. This seems to be create bottlenecks.
Or you use Raid 4 only for OS?

Also, what are the VMs doing? Are they doing a lot of IO inside? 50 VMs on 4 disks seems a little much also if each VM do many IO. (If you use the 4TB SSD for storage)
Lastly, are the disk volumes thin provision or thick?

All this could help you figure out some more things I think.

Kind regards,
T

On 4 Jun 2018, at 22:11, Thomas Fecke <thomas.fecke@eset.de> wrote:

Hey Juan,
 
That would be perfect. Ill searching for weeks now and can´t find the Bottleneck
 
The Storage is attached to an 10 gig switch
Storage:
 
1HE 19'' Chassis mit 4 hot-swap Einschüben
600W Platinum PSU
Intel Xeon E5-2620 v4 CPU
16GB DDR4-2400 regECC RAM (2x8)
4x 1TB SATA3 SSD (Samsung Pro)
LSI3108 Raidcontroller
2x Intel 10G-BaseT LAN
1x ded. KVM Port (IPMI2.0)
 
 
The Storage is attached to an 10 gig Switch. Just our Hypervisors are connected aswell to that switch. I don’t know the Switch Model, its rented from our Hoster ( like the Hardware ).
 
The Data Domain is shared via NFS.
 
We are working a lot with Templates – so a single Template gets deployed like 20 Times. Dunno if its important. The Guests running Win 10 and Win Server 2016 – Guest tools are installed. 
 
“Iotop” show about 100 MB/Sec
 
“Io sta”t show Storage Timeouts
 
dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=sync
1+0 Datensätze ein
1+0 Datensätze aus
1073741824 Bytes (1,1 GB) kopiert, 10,7527 s, 99,9 MB/s
 
DD ist way to low. When I reboot the Server its up to 800-900 MB/s. It drops slowly to under 100 in about 5 Minutes. Like a Cache that is filling up.
 
RAM and CPU are fine ( maximum 50% system load – average 30% ).
 
File system is XFS – Raid 4 is used
 
 
 
From: Juan Pablo <pablo.localhost@gmail.com> 
Sent: Montag, 4. Juni 2018 21:10
To: Thomas Fecke <thomas.fecke@eset.de>
Cc: users@ovirt.org
Subject: Re: [ovirt-users] Storage IO
 
Hi Thomas, so you are seeing high load on your storage and you are asking 'why'? an answer with the facts you give would be: you are using your storage, so, you have storage IO.
 
so, if you want to dive deeper:
-which storage are you using, specs would be nice.
-which host model are you using?
-network specs? card model, etc. switch model, etc.
 
hows your setup made? iscsi? nfs? gluster?
 
based on the former, we might get a better idea and after this some tests could be made if needed to find if there's a bottleneck or if the environment is working as expected..
 
regards,  
 
 
2018-06-04 14:29 GMT-03:00 Thomas Fecke <thomas.fecke@eset.de>:
Hey Guys,
 
sorry i need to ask again.
 
We got 2 Hypervisor with about 50 running VM´s and a single Storage with 10 Gig connection.
 
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               3,00   694,00 1627,00  947,00 103812,00 61208,00   128,22     6,78    2,63    2,13    3,49   0,39  99,70
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    3,70   31,37    0,00   64,93
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               1,00   805,00  836,00  997,00 43916,00 57900,00   111,09     6,00    3,27    1,87    4,44   0,54  99,30
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    3,54   29,96    0,00   66,50
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               2,00   822,00 1160,00 1170,00 46700,00 52176,00    84,87     5,68    2,44    1,57    3,30   0,43  99,50
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    5,05   31,46    0,00   63,50
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               3,00  1248,00 2337,00 1502,00 134932,00 48536,00    95,58     6,59    1,72    1,53    2,01   0,26  99,30
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    3,95   31,79    0,00   64,26
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00   704,00  556,00 1292,00 19908,00 72600,00   100,12     5,50    2,99    1,83    3,48   0,54  99,50
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    3,03   28,90    0,00   68,07
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00   544,00  278,00 1095,00  7848,00 66124,00   107,75     5,31    3,87    1,49    4,47   0,72  99,10
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    3,03   29,32    0,00   67,65
 
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00   464,00  229,00 1172,00  6588,00 72384,00   112,74     5,44    3,88    1,67    4,31   0,71  99,50
 
 
 
 
and this is our Problem. Anyone know why our Storage recive that much of Precesses?
 
Thanks in advance


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YENR6R4ESX3JCOS7DYA354EOPNL6WGUN/

 
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VU3OMW456MUM2SCWEERGQBJIPCVCB57N/