<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 15, 2017 at 7:40 PM, Jim Kusznir <span dir="ltr"><<a href="mailto:jim@palousetech.com" target="_blank">jim@palousetech.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr">I tried to create a gluster volume on the georep node by running:<div><br></div><div>gluster volume create engine-rep replica 1 georep.nwfiber.com:/mnt/gluste<wbr>r/engine-rep</div><div><br></div><div>I got back an error saying replica must be > 1. So I tried to create it again:</div></div></blockquote><div><br></div><div>"replica 1" is not required on command when you're not replicating to another server.<br></div><div>So,<br><br><div>gluster volume create engine-rep georep.nwfiber.com:/mnt/gluste<wbr>r/engine-rep</div><div><br></div><br><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div><br></div><div>gluster volume create engine-rep replica 2 georep.nwfiber.com:/mnt/gluste<wbr>r/engine-rep server2.nwfiber.com:/mnt/glust<wbr>er/engine-rep<br></div><div><br></div><div>where server2 did not exist. That failed too, but I don't recall the error message.</div><div><br></div><div>gluster is installed, but when I try and start it with the init script, it fails to start with a complaint about reading the block file; my googling indicated that's the error you get until you've created a gluster volume, and that was the first clue to me that maybe I needed to create one first.</div><div><br></div><div>So, how do I create a replica 1 volume?</div><div><br></div><div><br></div><div>Thinking way ahead, I have a related replica question: Currently my ovirt nodes are also my gluster nodes (replica 2 arbitrar 1). Eventually I'll want to pull my gluster off onto dedicated hardware I suspect. If I do so, do I need 3 servers, or is a replica 2 sufficient? I guess I could have an ovirt node continue to be an arbitrar... I would eventually like to distribute my ovirt cluster accross multiple locations with the option for remote failover (say location A looses all its network and/or power; have important VMs started at location B in addition to location B's normal VMs). I assume at this point the recommended arch would be:</div><div><br></div><div>2 Gluster servers at each location</div><div>Each location has a gluster volume for that location, and is georep for the other location (so all my data will physically exist on 4 gluster servers). I probably won't have more than 2 or 3 ovirt hosts at each location, so I don't expect this to be a "heavy use" system.</div></div></blockquote><div><br></div><div>To move to a dedicated gluster setup, 2 gluster servers + 1 oVirt node holding the arbiter brick should work.<br><br></div><div>Distributed or stretch cluster does not work off the bat yet. If a volume is geo-replicated to another location, the destination volume cannot be a storage domain in a cluster, as geo-replication needs the destination volume data to be in sync with master. <br>Another option is for the master volume/cluster is spread across geographies - but this requires the network latency to be very low in order for this to work as all operations are synchronous in gluster.<br>We're working on features in both gluster and Ovirt to support the stretched cluster requirements. <br><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div><br></div><div>Am I on track? I'd be interested to learn what others suggest for this deployment model.</div></div><div class="m_-7382756092833615085gmail-HOEnZb"><div class="m_-7382756092833615085gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, May 14, 2017 at 11:09 PM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr">Adding Aravinda<br><div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-7382756092833615085gmail-m_2217706634413760133h5">On Sat, May 13, 2017 at 11:21 PM, Jim Kusznir <span dir="ltr"><<a href="mailto:jim@palousetech.com" target="_blank">jim@palousetech.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi All:<div><br></div><div>I've been trying to set up georeplication for a while now, but can't seem to make it work. I've found documentation on the web (mostly <a href="https://gluster.readthedocs.io/en/refactor/Administrator%20Guide/Geo%20Replication/" target="_blank">https://gluster.readth<wbr>edocs.io/en/refactor/Administr<wbr>ator%20Guide/Geo%20Replication<wbr>/</a>), and I found <a href="http://blog.gluster.org/2015/09/introducing-georepsetup-gluster-geo-replication-setup-tool/" target="_blank">http://blog.gluster.org/<wbr>2015/09/introducing-georepsetu<wbr>p-gluster-geo-replication-setu<wbr>p-tool/</a></div><div><br></div><div>Unfortunately, it seems that some critical steps are missing from both, and I can't figure out for sure what they are.</div><div><br></div><div>My environment:</div><div><br></div><div>Production: replica 2 + arbitrator running on my 3-node oVirt cluster, 3 volumes (engine, data, iso).</div><div><br></div><div>New geo-replication: Raspberry Pi3 with USB hard drive shoved in some other data closet off-site.</div><div><br></div><div>I've installed rasbian-lite, and after much fighting, got glusterfs-*-3.8.11 installed. I've created my mountpoint (USB hard drive, much larger than my gluster volumes), and then ran the command. I get this far:</div><div><br></div><div><div>[ OK] <a href="http://georep.nwfiber.com" target="_blank">georep.nwfiber.com</a> is Reachable(Port 22)</div><div>[ OK] SSH Connection established <a href="mailto:root@georep.nwfiber.com" target="_blank">root@georep.nwfiber.com</a></div><div>[ OK] Master Volume and Slave Volume are compatible (Version: 3.8.11)</div><div>[NOT OK] Unable to Mount Gluster Volume georep.nwfiber.com:engine-rep</div></div><div><br></div><div>Trying it with the steps in the gluster docs also has the same problem. No long files are generated on the slave. Log files on the master include: </div><div><br></div><div><div>[root@ovirt1 geo-replication]# more georepsetup.mount.log</div><div>[2017-05-13 17:26:27.318599] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-glusterfs: Started running glusterfs version 3.8.11 (args:</div><div> glusterfs --xlator-option="*dht.lookup-u<wbr>nhashed=off" --volfile-server localhost --volfile-id engine -l /var/log/glusterfs/geo-repli</div><div>cation/georepsetup.mount.log --client-pid=-1 /tmp/georepsetup_wZtfkN)</div><div>[2017-05-13 17:26:27.341170] I [MSGID: 101190] [event-epoll.c:628:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2017-05-13 17:26:27.341260] E [socket.c:2309:socket_connect_<wbr>finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused</div><div>)</div><div>[2017-05-13 17:26:27.341846] E [glusterfsd-mgmt.c:1908:mgmt_r<wbr>pc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: local</div><div>host (Transport endpoint is not connected)</div><div>[2017-05-13 17:26:31.335849] I [MSGID: 101190] [event-epoll.c:628:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 2</div><div>[2017-05-13 17:26:31.337545] I [MSGID: 114020] [client.c:2356:notify] 0-engine-client-0: parent translators are ready, attempting co</div><div>nnect on transport</div><div>[2017-05-13 17:26:31.344485] I [MSGID: 114020] [client.c:2356:notify] 0-engine-client-1: parent translators are ready, attempting co</div><div>nnect on transport</div><div>[2017-05-13 17:26:31.345146] I [rpc-clnt.c:1965:rpc_clnt_reco<wbr>nfig] 0-engine-client-0: changing port to 49157 (from 0)</div><div>[2017-05-13 17:26:31.350868] I [MSGID: 114020] [client.c:2356:notify] 0-engine-client-2: parent translators are ready, attempting co</div><div>nnect on transport</div><div>[2017-05-13 17:26:31.355946] I [MSGID: 114057] [client-handshake.c:1440:selec<wbr>t_server_supported_programs] 0-engine-client-0: Using P</div><div>rogram GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2017-05-13 17:26:31.356280] I [rpc-clnt.c:1965:rpc_clnt_reco<wbr>nfig] 0-engine-client-1: changing port to 49157 (from 0)</div><div>Final graph:</div><div>+-----------------------------<wbr>------------------------------<wbr>-------------------+</div><div> 1: volume engine-client-0</div><div> 2: type protocol/client</div><div> 3: option clnt-lk-version 1</div><div> 4: option volfile-checksum 0</div><div> 5: option volfile-key engine</div><div> 6: option client-version 3.8.11</div><div> 7: option process-uuid ovirt1.nwfiber.com-25660-2017/<wbr>05/13-17:26:27:311929-engine-c<wbr>lient-0-0-0</div><div> 8: option fops-version 1298437</div><div> 9: option ping-timeout 30</div><div> 10: option remote-host <a href="http://ovirt1.nwfiber.com" target="_blank">ovirt1.nwfiber.com</a></div><div> 11: option remote-subvolume /gluster/brick1/engine</div><div> 12: option transport-type socket</div><div> 13: option username 028984cf-0399-42e6-b04b-bb9b16<wbr>85c536</div><div> 14: option password eae737cc-9659-405f-865e-9a7ef9<wbr>7a3307</div><div> 15: option filter-O_DIRECT off</div><div> 16: option send-gids true</div><div> 17: end-volume</div><div> 18:</div><div> 19: volume engine-client-1</div><div> 20: type protocol/client</div><div> 21: option ping-timeout 30</div><div> 22: option remote-host <a href="http://ovirt2.nwfiber.com" target="_blank">ovirt2.nwfiber.com</a></div><div> 23: option remote-subvolume /gluster/brick1/engine</div><div> 24: option transport-type socket</div><div> 25: option username 028984cf-0399-42e6-b04b-bb9b16<wbr>85c536</div><div> 26: option password eae737cc-9659-405f-865e-9a7ef9<wbr>7a3307</div><div> 27: option filter-O_DIRECT off</div><div> 28: option send-gids true</div><div> 29: end-volume</div><div> 30:</div><div> 31: volume engine-client-2</div><div> 32: type protocol/client</div><div> 33: option ping-timeout 30</div><div> 34: option remote-host <a href="http://ovirt3.nwfiber.com" target="_blank">ovirt3.nwfiber.com</a></div><div> 35: option remote-subvolume /gluster/brick1/engine</div><div> 36: option transport-type socket</div><div> 37: option username 028984cf-0399-42e6-b04b-bb9b16<wbr>85c536</div><div> 38: option password eae737cc-9659-405f-865e-9a7ef9<wbr>7a3307</div><div> 39: option filter-O_DIRECT off</div><div> 40: option send-gids true</div><div> 41: end-volume</div><div> 42:</div><div> 43: volume engine-replicate-0</div><div> 44: type cluster/replicate</div><div> 45: option arbiter-count 1</div><div> 46: option data-self-heal-algorithm full</div><div> 47: option eager-lock enable</div><div> 48: option quorum-type auto</div><div> 49: option shd-max-threads 6</div><div> 50: option shd-wait-qlength 10000</div><div> 51: option locking-scheme granular</div><div> 52: subvolumes engine-client-0 engine-client-1 engine-client-2</div><div> 53: end-volume</div><div> 54:</div><div> 55: volume engine-dht</div><div> 56: type cluster/distribute</div><div> 57: option lock-migration off</div><div> 58: subvolumes engine-replicate-0</div><div> 59: end-volume</div><div> 60:</div><div> 61: volume engine-shard</div><div> 62: type features/shard</div><div> 63: option shard-block-size 512MB</div><div> 64: subvolumes engine-dht</div><div> 65: end-volume</div><div> 66:</div><div> 67: volume engine-write-behind</div><div> 68: type performance/write-behind</div><div> 69: option strict-O_DIRECT on</div><div> 70: subvolumes engine-shard</div><div> 71: end-volume</div><div> 72:</div><div> 73: volume engine-readdir-ahead</div><div> 74: type performance/readdir-ahead</div><div> 75: subvolumes engine-write-behind</div><div> 76: end-volume</div><div> 77:</div><div> 78: volume engine-open-behind</div><div> 79: type performance/open-behind</div><div> 80: subvolumes engine-readdir-ahead</div><div> 81: end-volume</div><div> 82:</div><div> 83: volume engine</div><div> 84: type debug/io-stats</div><div> 85: option log-level INFO</div><div> 86: option latency-measurement off</div><div> 87: option count-fop-hits off</div><div> 88: subvolumes engine-open-behind</div><div> 89: end-volume</div><div> 90:</div><div> 91: volume meta-autoload</div><div> 92: type meta</div><div> 93: subvolumes engine</div><div> 94: end-volume</div><div> 95:</div><div>+-----------------------------<wbr>------------------------------<wbr>-------------------+</div><div>[2017-05-13 17:26:31.360579] I [MSGID: 114046] [client-handshake.c:1216:clien<wbr>t_setvolume_cbk] 0-engine-client-0: Connected to engine</div><div>-client-0, attached to remote volume '/gluster/brick1/engine'.</div><div>[2017-05-13 17:26:31.360599] I [MSGID: 114047] [client-handshake.c:1227:clien<wbr>t_setvolume_cbk] 0-engine-client-0: Server and Client l</div><div>k-version numbers are not same, reopening the fds</div><div>[2017-05-13 17:26:31.360707] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-engine-replicate-0: Subvolume 'engine-client-0' came</div><div> back up; going online.</div><div>[2017-05-13 17:26:31.360793] I [MSGID: 114035] [client-handshake.c:202:client<wbr>_set_lk_version_cbk] 0-engine-client-0: Server lk versi</div><div>on = 1</div><div>[2017-05-13 17:26:31.361284] I [rpc-clnt.c:1965:rpc_clnt_reco<wbr>nfig] 0-engine-client-2: changing port to 49158 (from 0)</div><div>[2017-05-13 17:26:31.365070] I [MSGID: 114057] [client-handshake.c:1440:selec<wbr>t_server_supported_programs] 0-engine-client-1: Using P</div><div>rogram GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2017-05-13 17:26:31.365788] I [MSGID: 114046] [client-handshake.c:1216:clien<wbr>t_setvolume_cbk] 0-engine-client-1: Connected to engine</div><div>-client-1, attached to remote volume '/gluster/brick1/engine'.</div><div>[2017-05-13 17:26:31.365821] I [MSGID: 114047] [client-handshake.c:1227:clien<wbr>t_setvolume_cbk] 0-engine-client-1: Server and Client l</div><div>k-version numbers are not same, reopening the fds</div><div>[2017-05-13 17:26:31.366059] I [MSGID: 114035] [client-handshake.c:202:client<wbr>_set_lk_version_cbk] 0-engine-client-1: Server lk versi</div><div>on = 1</div><div>[2017-05-13 17:26:31.369948] I [MSGID: 114057] [client-handshake.c:1440:selec<wbr>t_server_supported_programs] 0-engine-client-2: Using P</div><div>rogram GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2017-05-13 17:26:31.370657] I [MSGID: 114046] [client-handshake.c:1216:clien<wbr>t_setvolume_cbk] 0-engine-client-2: Connected to engine</div><div>-client-2, attached to remote volume '/gluster/brick1/engine'.</div><div>[2017-05-13 17:26:31.370683] I [MSGID: 114047] [client-handshake.c:1227:clien<wbr>t_setvolume_cbk] 0-engine-client-2: Server and Client l</div><div>k-version numbers are not same, reopening the fds</div><div>[2017-05-13 17:26:31.383548] I [MSGID: 114035] [client-handshake.c:202:client<wbr>_set_lk_version_cbk] 0-engine-client-2: Server lk versi</div><div>on = 1</div><div>[2017-05-13 17:26:31.383649] I [fuse-bridge.c:4147:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 k</div><div>ernel 7.22</div><div>[2017-05-13 17:26:31.383676] I [fuse-bridge.c:4832:fuse_graph<wbr>_sync] 0-fuse: switched to graph 0</div><div>[2017-05-13 17:26:31.385453] I [MSGID: 108031] [afr-common.c:2157:afr_local_d<wbr>iscovery_cbk] 0-engine-replicate-0: selecting local rea</div><div>d_child engine-client-0</div><div>[2017-05-13 17:26:31.396741] I [fuse-bridge.c:5080:fuse_threa<wbr>d_proc] 0-fuse: unmounting /tmp/georepsetup_wZtfkN</div><div>[2017-05-13 17:26:31.397086] W [glusterfsd.c:1327:cleanup_and<wbr>_exit] (-->/lib64/libpthread.so.0(+0x<wbr>7dc5) [0x7f8838df6dc5] -->glusterf</div><div>s(glusterfs_sigwaiter+0xe5) [0x7f883a488cd5] -->glusterfs(cleanup_and_exit+<wbr>0x6b) [0x7f883a488b4b] ) 0-: received signum (15), shutti</div><div>ng down</div><div>[2017-05-13 17:26:31.397112] I [fuse-bridge.c:5788:fini] 0-fuse: Unmounting '/tmp/georepsetup_wZtfkN'.</div><div>[2017-05-13 17:26:31.413901] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-glusterfs: Started running glusterfs version 3.8.11 (args:</div><div> glusterfs --xlator-option="*dht.lookup-u<wbr>nhashed=off" --volfile-server <a href="http://georep.nwfiber.com" target="_blank">georep.nwfiber.com</a> --volfile-id engine -l /var/log/glusterfs/</div><div>geo-replication/georepsetup.mo<wbr>unt.log --client-pid=-1 /tmp/georepsetup_M5poIr)</div><div>[2017-05-13 17:26:31.458733] I [MSGID: 101190] [event-epoll.c:628:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2017-05-13 17:26:31.458833] E [socket.c:2309:socket_connect_<wbr>finish] 0-glusterfs: connection to <a href="http://192.168.8.126:24007" target="_blank">192.168.8.126:24007</a> failed (Connecti</div><div>on refused)</div><div>[2017-05-13 17:26:31.458886] E [glusterfsd-mgmt.c:1908:mgmt_r<wbr>pc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: geore</div><div><a href="http://p.nwfiber.com" target="_blank">p.nwfiber.com</a> (Transport endpoint is not connected)</div><div>[2017-05-13 17:26:31.458900] I [glusterfsd-mgmt.c:1926:mgmt_r<wbr>pc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers</div><div>[2017-05-13 17:26:31.459173] W [glusterfsd.c:1327:cleanup_and<wbr>_exit] (-->/lib64/libgfrpc.so.0(rpc_c<wbr>lnt_notify+0xdb) [0x7f18d6c89aab]</div><div>-->glusterfs(+0x10309) [0x7f18d73b9309] -->glusterfs(cleanup_and_exit+<wbr>0x6b) [0x7f18d73b2b4b] ) 0-: received signum (1), shutting dow</div><div>n</div><div>[2017-05-13 17:26:31.459218] I [fuse-bridge.c:5788:fini] 0-fuse: Unmounting '/tmp/georepsetup_M5poIr'.</div><div>[2017-05-13 17:26:31.459887] W [glusterfsd.c:1327:cleanup_and<wbr>_exit] (-->/lib64/libpthread.so.0(+0x<wbr>7dc5) [0x7f18d5d20dc5] -->glusterf</div><div>s(glusterfs_sigwaiter+0xe5) [0x7f18d73b2cd5] -->glusterfs(cleanup_and_exit+<wbr>0x6b) [0x7f18d73b2b4b] ) 0-: received signum (15), shutti</div><div>ng down</div></div><div><br></div><div>I don't know what to make of that.</div><div><br></div><div>On a whim, I thought that perhaps the georep setup does not set up the remote volume (I assumed it would, I thought that was what the ssh was required for, and none of the instructions mentioned create your destination (replication) volume. So I tried to create it, but it won't let me create a volume with replica 1. this is already a backup, I don't need a backup of a backup. This further supported my thought that the volume needs to be created by the georep setup commands.</div></div></blockquote><div><br></div></div></div><div>The destination or slave volume needs to be created prior to setting up the geo-replication session. You should be able to create a replica 1 volume as destination volume. How did you try to create this?<br><br></div><div>Is glusterd running on geore<a href="http://p.nwfiber.com" target="_blank">p.nwfiber.com</a> ? And are the gluster ports open?<br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><div dir="ltr"><div><br></div><div>Where am I wrong / what do I need to do to fix this?</div><span class="m_-7382756092833615085gmail-m_2217706634413760133m_-6734385470561030321gmail-HOEnZb"><font color="#888888"><div><br></div><div>--Jim</div></font></span></div>
<br></span>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>