<div dir="ltr">Here's gluster volume info:<div><br></div><div><div>[root@ovirt2 ~]# gluster volume info</div><div> </div><div>Volume Name: data</div><div>Type: Replicate</div><div>Volume ID: e670c488-ac16-4dd1-8bd3-e43b2e42cc59</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x (2 + 1) = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: ovirt1.nwfiber.com:/gluster/brick2/data</div><div>Brick2: ovirt2.nwfiber.com:/gluster/brick2/data</div><div>Brick3: ovirt3.nwfiber.com:/gluster/brick2/data (arbiter)</div><div>Options Reconfigured:</div><div>changelog.changelog: on</div><div>geo-replication.ignore-pid-check: on</div><div>geo-replication.indexing: on</div><div>server.allow-insecure: on</div><div>performance.readdir-ahead: on</div><div>performance.quick-read: off</div><div>performance.read-ahead: off</div><div>performance.io-cache: off</div><div>performance.stat-prefetch: off</div><div>cluster.eager-lock: enable</div><div>network.remote-dio: enable</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>features.shard: on</div><div>features.shard-block-size: 512MB</div><div>performance.low-prio-threads: 32</div><div>cluster.data-self-heal-algorithm: full</div><div>cluster.locking-scheme: granular</div><div>cluster.shd-wait-qlength: 10000</div><div>cluster.shd-max-threads: 8</div><div>network.ping-timeout: 30</div><div>user.cifs: off</div><div>nfs.disable: on</div><div>performance.strict-o-direct: on</div><div> </div><div>Volume Name: data-hdd</div><div>Type: Replicate</div><div>Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 172.172.1.11:/gluster/brick3/data-hdd</div><div>Brick2: 172.172.1.12:/gluster/brick3/data-hdd</div><div>Brick3: 172.172.1.13:/gluster/brick3/data-hdd</div><div>Options Reconfigured:</div><div>changelog.changelog: on</div><div>geo-replication.ignore-pid-check: on</div><div>geo-replication.indexing: on</div><div>transport.address-family: inet</div><div>performance.readdir-ahead: on</div><div> </div><div>Volume Name: engine</div><div>Type: Replicate</div><div>Volume ID: 87ad86b9-d88b-457e-ba21-5d3173c612de</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x (2 + 1) = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: ovirt1.nwfiber.com:/gluster/brick1/engine</div><div>Brick2: ovirt2.nwfiber.com:/gluster/brick1/engine</div><div>Brick3: ovirt3.nwfiber.com:/gluster/brick1/engine (arbiter)</div><div>Options Reconfigured:</div><div>changelog.changelog: on</div><div>geo-replication.ignore-pid-check: on</div><div>geo-replication.indexing: on</div><div>performance.readdir-ahead: on</div><div>performance.quick-read: off</div><div>performance.read-ahead: off</div><div>performance.io-cache: off</div><div>performance.stat-prefetch: off</div><div>cluster.eager-lock: enable</div><div>network.remote-dio: off</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>features.shard: on</div><div>features.shard-block-size: 512MB</div><div>performance.low-prio-threads: 32</div><div>cluster.data-self-heal-algorithm: full</div><div>cluster.locking-scheme: granular</div><div>cluster.shd-wait-qlength: 10000</div><div>cluster.shd-max-threads: 6</div><div>network.ping-timeout: 30</div><div>user.cifs: off</div><div>nfs.disable: on</div><div>performance.strict-o-direct: on</div><div> </div><div>Volume Name: iso</div><div>Type: Replicate</div><div>Volume ID: b1ba15f5-0f0f-4411-89d0-595179f02b92</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x (2 + 1) = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: ovirt1.nwfiber.com:/gluster/brick4/iso</div><div>Brick2: ovirt2.nwfiber.com:/gluster/brick4/iso</div><div>Brick3: ovirt3.nwfiber.com:/gluster/brick4/iso (arbiter)</div><div>Options Reconfigured:</div><div>performance.readdir-ahead: on</div><div>performance.quick-read: off</div><div>performance.read-ahead: off</div><div>performance.io-cache: off</div><div>performance.stat-prefetch: off</div><div>cluster.eager-lock: enable</div><div>network.remote-dio: off</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>features.shard: on</div><div>features.shard-block-size: 512MB</div><div>performance.low-prio-threads: 32</div><div>cluster.data-self-heal-algorithm: full</div><div>cluster.locking-scheme: granular</div><div>cluster.shd-wait-qlength: 10000</div><div>cluster.shd-max-threads: 6</div><div>network.ping-timeout: 30</div><div>user.cifs: off</div><div>nfs.disable: on</div><div>performance.strict-o-direct: on</div></div><div><br></div><div>--------------</div><div><br></div><div>When I try and turn on profiling, I get:</div><div><br></div><div><div>[root@ovirt2 ~]# gluster volume profile data-hdd start</div><div>Another transaction is in progress for data-hdd. Please try again after sometime.</div></div><div><br></div><div>I don't know what that other transaction is, but I am having some "odd behavior" this morning, like a vm disk move between data and data-hdd that stuck at 84% overnight.</div><div><br></div><div>I've been asking on IRC how to "un-stick" this transfer, as the VM cannot be started, and I can't seem to do anything about it.</div><div><br></div><div>--Jim</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 19, 2018 at 2:14 AM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Mar 19, 2018 at 7:39 AM, Jim Kusznir <span dir="ltr"><<a href="mailto:jim@palousetech.com" target="_blank">jim@palousetech.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello:<div><br></div><div>This past week, I created a new gluster store, as I was running out of disk space on my main, SSD-backed storage pool. I used 2TB Seagate FireCuda drives (hybrid SSD/spinning). Hardware is Dell R610's with integral PERC/6i cards. I placed one disk per machine, exported the disk as a single disk volume from the raid controller, formatted it XFS, mounted it, and dedicated it to a new replica 3 gluster volume.</div><div><br></div><div>Since doing so, I've been having major performance problems. One of my windows VMs sits at 100% disk utilization nearly continously, and its painful to do anything on it. A Zabbix install on CentOS using mysql as the backing has 70%+ iowait nearly all the time, and I can't seem to get graphs loaded from the web console. Its also always spewing errors that ultimately come down to insufficient disk performance issues.</div><div><br></div><div>All of this was working OK before the changes. There are two:</div><div><br></div><div>Old storage was SSD backed, Replica 2 + arb, and running on the same GigE network as management and main VM network.</div><div><br></div><div>New storage was created using the dedicated Gluster network (running on em4 on these servers, completely different subnet (174.x vs 192.x), and was created replica 3 (no arb), on the FireCuda disks (seem to be the fastest I could afford for non-SSD, as I needed a lot more storage).</div><div><br></div><div>My attempts to watch so far have NOT shown maxed network interfaces (using bwm-ng on the command line); in fact, the gluster interface is usually below 20% utilized.</div><div><br></div><div>I'm not sure how to meaningfully measure the performance of the disk itself; I'm not sure what else to look at. My cluster is not very usable currently, though. IOWait on my hosts appears to be below 0.5%, usually 0.0 to 0.1. Inside the VMs is a whole different story.</div><div><br></div><div>My cluster is currently running ovirt 4.1. I'm interested in going to 4.2, but I think I need to fix this first.</div></div></blockquote><div><br></div><div><br></div></span><div>Can you provide the info of the volume using "gluster volume info" and also profile the volume while running the tests where you experience the performance issue, and share results?</div><div><br></div><div>For info on how to profile (server-side profiling) - <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/" target="_blank">https://docs.gluster.org/en/<wbr>latest/Administrator%20Guide/<wbr>Performance%20Testing/</a><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Thanks!</div><span class="m_4050124923047810885gmail-HOEnZb"><font color="#888888"><div>--Jim</div></font></span></div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>