glusterfs takes a long time to syncing after host has been rebooted
Hi. I am testing oVirt 3.5.2 with 3 hosts (Dell R210). Storage type is GlusterFS (replicate 3): each host has a single 3TB HDD. When I put one host into maintence mode, then reboot it and after system has been started, the glusterfsd proccess takes a long time (more then hours with gigabyte network) to syncing. It seems to be reload all data instead of download only changes . I have several VMs, but only one VM was running (postfix relay) at that moment, so there were no lots of changes on gluster volume. I've googled for this issue without success. Is it normal situation? Or what can I do to resolve the problem? I can provide any additional info. Thanks.
--_000_D188D7F1CAE1soerenmalchowmconnet_ Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: quoted-printable Hi, This is strictly speaking a gluster issue, not an ovirt issue, however, fir= st of all, it does take very long to resync, that is normal, second, we hav= e vey good experiences so far in tuning a few of the parameters in gluster These are the ones (at least partly) we modified cluster.data-self-heal-algorithm diff cluster.background-self-heal-count 16 performance.io-thread-count 32 performance.high-prio-threads 24 performance.normal-prio-threads 24 performance.low-prio-threads 16 performance.least-prio-threads 4 Check #> gluster colume get VOLUMENAME all For more And even after optimization it will still take very long Regards Soeren From: =DE=F0=E8=E9 =CF=EE=EB=F2=EE=F0=E0=F6=EA=E8=E9 <y.poltoratskiy@gmail.= com<mailto:y.poltoratskiy@gmail.com>> Date: Monday 25 May 2015 12:46 To: "users@ovirt.org<mailto:users@ovirt.org>" <users@ovirt.org<mailto:users= @ovirt.org>> Subject: [ovirt-users] glusterfs takes a long time to syncing after host ha= s been rebooted Hi. I am testing oVirt 3.5.2 with 3 hosts (Dell R210). Storage type is GlusterF= S (replicate 3): each host has a single 3TB HDD. When I put one host into m= aintence mode, then reboot it and after system has been started, the gluste= rfsd proccess takes a long time (more then hours with gigabyte network) to = syncing. It seems to be reload all data instead of download only changes . = I have several VMs, but only one VM was running (postfix relay) at that mom= ent, so there were no lots of changes on gluster volume. I've googled for this issue without success. Is it normal situation? Or what can I do to resolve the problem? I can prov= ide any additional info. Thanks. --_000_D188D7F1CAE1soerenmalchowmconnet_ Content-Type: text/html; charset="windows-1251" Content-ID: <0984B64D6ACCCA458A605AF18F4B9F27@liquidcampaign.com> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-1= 251"> </head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami= ly: Calibri, sans-serif;"> <div>Hi,</div> <div><br> </div> <div>This is strictly speaking a gluster issue, not an ovirt issue, however= , first of all, it does take very long to resync, that is normal, second, w= e have vey good experiences so far in tuning a few of the parameters in glu= ster</div> <div><br> </div> <div>These are the ones (at least partly) we modified</div> <div><br> </div> <div>cluster.data-self-heal-algorithm diff</div> <div>cluster.background-self-heal-count 16</div> <div> <div>performance.io-thread-count = 32</div> <div>performance.high-prio-threads 24</d= iv> <div>performance.normal-prio-threads 24</div> <div>performance.low-prio-threads = 16</div> <div>performance.least-prio-threads 4</di= v> </div> <div><br> </div> <div><br> </div> <div>Check </div> <div><br> </div> <div>#> gluster colume get VOLUMENAME all</div> <div><br> </div> <div>For more</div> <div><br> </div> <div>And even after optimization it will still take very long</div> <div><br> </div> <div>Regards</div> <div>Soeren</div> <div><br> </div> <div><br> </div> <div><br> </div> <span id=3D"OLK_SRC_BODY_SECTION"> <div style=3D"font-family:Calibri; font-size:11pt; text-align:left; color:b= lack; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM:= 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid;= BORDER-RIGHT: medium none; PADDING-TOP: 3pt"> <span style=3D"font-weight:bold">From: </span>=DE=F0=E8=E9 =CF=EE=EB=F2=EE= =F0=E0=F6=EA=E8=E9 <<a href=3D"mailto:y.poltoratskiy@gmail.com">y.poltor= atskiy@gmail.com</a>><br> <span style=3D"font-weight:bold">Date: </span>Monday 25 May 2015 12:46<br> <span style=3D"font-weight:bold">To: </span>"<a href=3D"mailto:users@o= virt.org">users@ovirt.org</a>" <<a href=3D"mailto:users@ovirt.org">= users@ovirt.org</a>><br> <span style=3D"font-weight:bold">Subject: </span>[ovirt-users] glusterfs ta= kes a long time to syncing after host has been rebooted<br> </div> <div><br> </div> <div> <div> <div dir=3D"ltr">Hi.<br> <div><br> I am testing oVirt 3.5.2 with 3 hosts (Dell R210). Storage type is GlusterF= S (replicate 3): each host has a single 3TB HDD. When I put one host into m= aintence mode, then reboot it and after system has been started, the gluste= rfsd proccess takes a long time (more then hours with gigabyte network) to syncing. It seems to be reload = all data instead of download only changes . I have several VMs, but only on= e VM was running (postfix relay) at that moment, so there were no lots of c= hanges on gluster volume. <br> <br> I've googled for this issue without success. <br> <br> Is it normal situation? Or what can I do to resolve the problem? I can prov= ide any additional info.<br> <br> Thanks.<br> </div> </div> </div> </div> </span> </body> </html> --_000_D188D7F1CAE1soerenmalchowmconnet_--
Hi, i have the same setup and "problem". Nothing really happens on my SAN Network interface. I can only see with iotop a read rate of about 50MB/sec on every node. So i wonder if it tries to heal/verify checksums of files. Maybe it does that so slow to prevent over stressing the I/O of the disk?! I really dunno. (yes its a gluster issue, but i still think we should care about it) Mario Am 25.05.15 um 12:46 schrieb Юрий Полторацкий:
Hi.
I am testing oVirt 3.5.2 with 3 hosts (Dell R210). Storage type is GlusterFS (replicate 3): each host has a single 3TB HDD. When I put one host into maintence mode, then reboot it and after system has been started, the glusterfsd proccess takes a long time (more then hours with gigabyte network) to syncing. It seems to be reload all data instead of download only changes . I have several VMs, but only one VM was running (postfix relay) at that moment, so there were no lots of changes on gluster volume.
I've googled for this issue without success.
Is it normal situation? Or what can I do to resolve the problem? I can provide any additional info.
Thanks.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
ml＠ohnewald.net -
Soeren Malchow -
Юрий Полторацкий