<div dir="ltr">Thank you for publishing your results. It's quite encouraging and helps us to push to make gfapi access the default in upcoming release.<br><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 11:58 PM, Darrell Budic <span dir="ltr"><<a href="mailto:budic@onholyground.com" target="_blank">budic@onholyground.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Ok, so only for the HA engine eh? Been meaning to ask about that, since my hosted engine wasn’t using it. Tolerable, live disk migrations are way less valuable to me than better disk performance :) Other comments inline: <br><div><blockquote type="cite"><hr style="border:none;border-top:solid #b5c4df 1.0pt;padding:0 0 0 0;margin:10px 0 5px 0"><span id="m_561129319840432143RwhHeaderAttributes"><font style="font:13.0px Helvetica;color:#000000" size="4" color="#000000" face="Helvetica"><b>From:</b> Sahina Bose <<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></font></span><span class=""><br>
<span><font style="font:13.0px Helvetica;color:#000000" size="4" color="#000000" face="Helvetica"><b>Subject:</b> Re: [ovirt-users] [ANN] oVirt 4.1.9 Release is now available</font></span><br>
</span><span><font style="font:13.0px Helvetica;color:#000000" size="4" color="#000000" face="Helvetica"><b>Date:</b> January 25, 2018 at 12:24:34 AM CST</font></span><br>
<span><font style="font:13.0px Helvetica;color:#000000" size="4" color="#000000" face="Helvetica"><b>To:</b> Darrell Budic</font></span><br>
<span><font style="font:13.0px Helvetica;color:#000000" size="4" color="#000000" face="Helvetica"><b>Cc:</b> Lev Veyde; users</font></span><span class=""><br>
<br class="m_561129319840432143Apple-interchange-newline"><div>- it doesn’t seem to affect my HA vms, I’ve seen my 4.1.8 system properly restart systems using it (node/libvirtd crash that seems to have been related to spectre/meltdown firmwares)<br><div class="gmail_quote" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div><br></div><div>HA is an issue only when the gluster server used to provide volume information is down. For instance, if you have provided the "serverA:/volumeA" in your storage domain path , and if the other servers in replica are up but serverA is down, VM cannot restart. Have you tested this?</div></div></div></span></blockquote><div><br></div><div>I’m using a DNS based methods & backup server mount options to ensure the volume info is always available, so this particular problem shouldn’t affect me (and hasn’t in my testing).</div><span class=""><br><blockquote type="cite"><div><div class="gmail_quote" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div><br></div><div>Can you share any performance improvement numbers that you have seen after turning on libgfapi access, also information about your workload would be helpful.</div><div><br></div><div>thanks!<br></div></div></div></blockquote><br></span></div><div>Some not-very-scientific testing (can’t arrange dedicated disk time on this system) on a VM that hadn’t been covered yet gives me:</div><div><br></div><div>Before using gfapi:</div><div><br></div><div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">]# dd if=/dev/urandom of=test.file bs=1M count=1024</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1024+0 records in</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1024+0 records out</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1073741824 bytes (1.1 GB) copied, 90.1843 s, 11.9 MB/s</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># echo 3 > /proc/sys/vm/drop_caches</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># dd if=test.file of=/dev/null </span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">2097152+0 records in</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">2097152+0 records out</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1073741824 bytes (1.1 GB) copied, 3.94715 s, 272 MB/s</span></div></div><div><br></div></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># hdparm -tT /dev/vda</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;min-height:18px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">/dev/vda:</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"> Timing cached reads: 17322 MB in 2.00 seconds = 8673.49 MB/sec</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"> Timing buffered disk reads: 996 MB in 3.00 seconds = 331.97 MB/sec</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">#</span>bonnie++ -d . -s 8G -n 0 -m pre-glapi -f -b -u root</div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="color:rgb(0,200,255);font-family:"Fantasque Sans Mono";font-size:15px;background-color:rgba(0,0,0,0.901961)"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">pre-glapi 8G 196245 30 105331 15 962775 49 1638 34</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Latency 1578ms 1383ms 201ms 301ms</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">pre-glapi 8G 155937 27 102899 14 1030285 54 1763 45</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Latency 694ms 1333ms 114ms 229ms</span></div><div><br></div><div>(note, sequential reads seem to have been influenced by caching somewhere…)</div></span></div></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">After switching to gfapi:</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># dd if=/dev/urandom of=test.file bs=1M count=1024</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1024+0 records in</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1024+0 records out</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1073741824 bytes (1.1 GB) copied, 80.8317 s, 13.3 MB/s</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># echo 3 > /proc/sys/vm/drop_caches</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># dd if=test.file of=/dev/null </span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">2097152+0 records in</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">2097152+0 records out</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">1073741824 bytes (1.1 GB) copied, 3.3473 s, 321 MB/s</span></div></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># hdparm -tT /dev/vda</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;min-height:18px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">/dev/vda:</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"> Timing cached reads: 17112 MB in 2.00 seconds = 8568.86 MB/sec</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"> Timing buffered disk reads: 1406 MB in 3.01 seconds = 467.70 MB/sec</span></div></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">#bonnie++ -d . -s 8G -n 0 -m glapi -f -b -u root</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures"> glapi 8G 359100 59 185289 24 489575 31 2079 67</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Latency 160ms 355ms 36041us 185ms</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures"> glapi 8G 341307 57 180546 24 472572 35 2655 61</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Latency 153ms 394ms 101ms 116ms</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>So excellent improvement in write throughput, but the significant improvement in latency is what was most noticed by users. Anecdotal reports of 2x+ performance improvements, with one remarking that it’s like having dedicated disks :)</div><div><br></div></span></div></span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">This system is on my production cluster, so it’s not getting exclusive disk access, but this VM is not doing anything else itself. The cluster is 3 xeon </span>E5-2609 v3 @ 1.90GHz servers w/ 64G ram, SATA2 disks; 2 with 9x spindles each, 1 with 8x slightly faster disks (all spinners). Using ZFS stripes with lz4 compression and 10G connectivity to 8 hosts. Running gluster 3.12.3 at the moment. The cluster itself has about 70 running VMs in varying states of switching to gfapi use, but my main sql servers are using their own volumes and not competing for this one. These have not yet had the spectre/meltdown patches applied.</div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"> </span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures">This will be skewed because I forced it to not steal all the ram on the server (reads will certainly be cached), but an idea of what it can do disk wise, on the volume used above:</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-variant-ligatures:no-common-ligatures"># bonnie++ -d . -s 8G -n 0 -m zfs-server -f -b -u root -r 4096</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">zfs-server 8G 604940 79 510410 87 1393862 99 3164 91</span></div><div style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:"Fantasque Sans Mono";color:rgb(0,200,255);background-color:rgba(0,0,0,0.901961)"><span style="font-variant-ligatures:no-common-ligatures">Latency 99545us 100ms 247us 152ms</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">Just for fun from one of the servers showing base load and this testing:</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><img id="m_5611293198404321437412EE8A-222F-4005-B266-9CB5BA492E46" src="cid:B68D24F8-81A5-4825-B0DE-39E6D46FA3BA@spaanem.onholyground.com"></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></span></div></div></blockquote></div><br></div></div></div>