<div dir="ltr"><div><div><div><div><div>ok I will answer by my self:<br></div>yes gluster daemon is managed by vdms:)<br></div>and to recover lost config simply one should add &quot;force&quot; keyword <br>gluster volume create GluReplica replica 3 arbiter 1 transport TCP,RDMA 
10.10.10.44:/zclei22/01/glu 10.10.10.42:/zclei21/01/glu 
10.10.10.41:/zclei26/01/glu force <br><br></div>now everything is up an running !<br></div>one annoying thing is epel dependency in the zfs and conflicting ovirt...<br></div>every time one need to enable and then disable epel.<br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 1, 2017 at 5:33 PM, Arman Khalatyan <span dir="ltr">&lt;<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>ok Finally by single brick up and running so I can access to data.<br></div>Now the question is do we need to run glusterd daemon on startup? or it is managed by vdsmd?<br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 1, 2017 at 2:36 PM, Arman Khalatyan <span dir="ltr">&lt;<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>all folders /var/lib/glusterd/vols/ are empty <br></div>In the history of one of the servers I found the command how it was created:<br><br>gluster volume create GluReplica replica 3 arbiter 1 transport TCP,RDMA 10.10.10.44:/zclei22/01/glu 10.10.10.42:/zclei21/01/glu 10.10.10.41:/zclei26/01/glu<br><br></div>But executing this command it claims that:<br>volume create: GluReplica: failed: /zclei22/01/glu is already part of a volume<br><br></div>Any chance to force it?<br><br><div><br></div></div><div class="m_690619254500226376HOEnZb"><div class="m_690619254500226376h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 1, 2017 at 12:13 PM, Ramesh Nachimuthu <span dir="ltr">&lt;<a href="mailto:rnachimu@redhat.com" target="_blank">rnachimu@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_690619254500226376m_5030294642598411805HOEnZb"><div class="m_690619254500226376m_5030294642598411805h5"><br>
<br>
<br>
<br>
----- Original Message -----<br>
&gt; From: &quot;Arman Khalatyan&quot; &lt;<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>&gt;<br>
&gt; To: &quot;users&quot; &lt;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&gt;<br>
&gt; Sent: Wednesday, March 1, 2017 3:10:38 PM<br>
&gt; Subject: Re: [ovirt-users] Gluster setup disappears any chance to recover?<br>
&gt;<br>
&gt; engine throws following errors:<br>
&gt; 2017-03-01 10:39:59,608+01 WARN<br>
&gt; [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector]<br>
&gt; (DefaultQuartzScheduler6) [d7f7d83] EVENT_ID:<br>
&gt; GLUSTER_VOLUME_DELETED_FROM_CL<wbr>I(4,027), Correlation ID: null, Call Stack:<br>
&gt; null, Custom Event ID: -1, Message: Detected deletion of volume GluReplica<br>
&gt; on cluster HaGLU, and deleted it from engine DB.<br>
&gt; 2017-03-01 10:39:59,610+01 ERROR<br>
&gt; [org.ovirt.engine.core.bll.glu<wbr>ster.GlusterSyncJob] (DefaultQuartzScheduler6)<br>
&gt; [d7f7d83] Error while removing volumes from database!:<br>
&gt; org.springframework.dao.DataIn<wbr>tegrityViolationException:<br>
&gt; CallableStatementCallback; SQL [{call deleteglustervolumesbyguids(?)<wbr>}];<br>
&gt; ERROR: update or delete on table &quot;gluster_volumes&quot; violates foreign key<br>
&gt; constraint &quot;fk_storage_connection_to_glus<wbr>tervolume&quot; on table<br>
&gt; &quot;storage_server_connections&quot;<br>
&gt; Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-<wbr>bd317623ed2d) is still referenced<br>
&gt; from table &quot;storage_server_connections&quot;.<br>
&gt; Where: SQL statement &quot;DELETE<br>
&gt; FROM gluster_volumes<br>
&gt; WHERE id IN (<br>
&gt; SELECT *<br>
&gt; FROM fnSplitterUuid(v_volume_ids)<br>
&gt; )&quot;<br>
&gt; PL/pgSQL function deleteglustervolumesbyguids(ch<wbr>aracter varying) line 3 at<br>
&gt; SQL statement; nested exception is org.postgresql.util.PSQLExcept<wbr>ion: ERROR:<br>
&gt; update or delete on table &quot;gluster_volumes&quot; violates foreign key constraint<br>
&gt; &quot;fk_storage_connection_to_glus<wbr>tervolume&quot; on table<br>
&gt; &quot;storage_server_connections&quot;<br>
&gt; Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-<wbr>bd317623ed2d) is still referenced<br>
&gt; from table &quot;storage_server_connections&quot;.<br>
&gt; Where: SQL statement &quot;DELETE<br>
&gt; FROM gluster_volumes<br>
&gt; WHERE id IN (<br>
&gt; SELECT *<br>
&gt; FROM fnSplitterUuid(v_volume_ids)<br>
&gt; )&quot;<br>
&gt; PL/pgSQL function deleteglustervolumesbyguids(ch<wbr>aracter varying) line 3 at<br>
&gt; SQL statement<br>
&gt; at<br>
&gt; org.springframework.jdbc.suppo<wbr>rt.SQLErrorCodeSQLExceptionTra<wbr>nslator.doTranslate(SQLErrorCo<wbr>deSQLExceptionTranslator.java:<wbr>243)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at<br>
&gt; org.springframework.jdbc.suppo<wbr>rt.AbstractFallbackSQLExceptio<wbr>nTranslator.translate(Abstract<wbr>FallbackSQLExceptionTranslator<wbr>.java:73)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at org.springframework.jdbc.core.<wbr>JdbcTemplate.execute(JdbcTempl<wbr>ate.java:1094)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at org.springframework.jdbc.core.<wbr>JdbcTemplate.call(JdbcTemplate<wbr>.java:1130)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at<br>
&gt; org.springframework.jdbc.core.<wbr>simple.AbstractJdbcCall.execut<wbr>eCallInternal(AbstractJdbcCall<wbr>.java:405)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at<br>
&gt; org.springframework.jdbc.core.<wbr>simple.AbstractJdbcCall.doExec<wbr>ute(AbstractJdbcCall.java:365)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at<br>
&gt; org.springframework.jdbc.core.<wbr>simple.SimpleJdbcCall.execute(<wbr>SimpleJdbcCall.java:198)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.dal.dbbr<wbr>oker.SimpleJdbcCallsHandler.ex<wbr>ecuteImpl(SimpleJdbcCallsHandl<wbr>er.java:135)<br>
&gt; [dal.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.dal.dbbr<wbr>oker.SimpleJdbcCallsHandler.ex<wbr>ecuteImpl(SimpleJdbcCallsHandl<wbr>er.java:130)<br>
&gt; [dal.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.dal.dbbr<wbr>oker.SimpleJdbcCallsHandler.ex<wbr>ecuteModification(SimpleJdbcCa<wbr>llsHandler.java:76)<br>
&gt; [dal.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.dao.glus<wbr>ter.GlusterVolumeDaoImpl.remov<wbr>eAll(GlusterVolumeDaoImpl.java<wbr>:233)<br>
&gt; [dal.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.bll.glus<wbr>ter.GlusterSyncJob.removeDelet<wbr>edVolumes(GlusterSyncJob.java:<wbr>521)<br>
&gt; [bll.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.bll.glus<wbr>ter.GlusterSyncJob.refreshVolu<wbr>meData(GlusterSyncJob.java:465<wbr>)<br>
&gt; [bll.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.bll.glus<wbr>ter.GlusterSyncJob.refreshClus<wbr>terData(GlusterSyncJob.java:13<wbr>3)<br>
&gt; [bll.jar:]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.bll.glus<wbr>ter.GlusterSyncJob.refreshLigh<wbr>tWeightData(GlusterSyncJob.jav<wbr>a:111)<br>
&gt; [bll.jar:]<br>
&gt; at sun.reflect.NativeMethodAccess<wbr>orImpl.invoke0(Native Method)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at<br>
&gt; sun.reflect.NativeMethodAccess<wbr>orImpl.invoke(NativeMethodAcce<wbr>ssorImpl.java:62)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at<br>
&gt; sun.reflect.DelegatingMethodAc<wbr>cessorImpl.invoke(DelegatingMe<wbr>thodAccessorImpl.java:43)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at java.lang.reflect.Method.invok<wbr>e(Method.java:498) [rt.jar:1.8.0_121]<br>
&gt; at<br>
&gt; org.ovirt.engine.core.utils.ti<wbr>mer.JobWrapper.invokeMethod(Jo<wbr>bWrapper.java:77)<br>
&gt; [scheduler.jar:]<br>
&gt; at org.ovirt.engine.core.utils.ti<wbr>mer.JobWrapper.execute(JobWrap<wbr>per.java:51)<br>
&gt; [scheduler.jar:]<br>
&gt; at <a href="http://org.quartz.core.JobRunShell.ru" target="_blank">org.quartz.core.JobRunShell.ru</a><wbr>n(JobRunShell.java:213) [quartz.jar:]<br>
&gt; at java.util.concurrent.Executors<wbr>$RunnableAdapter.call(Executor<wbr>s.java:511)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at java.util.concurrent.FutureTas<wbr>k.run(FutureTask.java:266)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at<br>
&gt; java.util.concurrent.ThreadPoo<wbr>lExecutor.runWorker(ThreadPool<wbr>Executor.java:1142)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at<br>
&gt; java.util.concurrent.ThreadPoo<wbr>lExecutor$Worker.run(ThreadPoo<wbr>lExecutor.java:617)<br>
&gt; [rt.jar:1.8.0_121]<br>
&gt; at java.lang.Thread.run(Thread.ja<wbr>va:745) [rt.jar:1.8.0_121]<br>
&gt; Caused by: org.postgresql.util.PSQLExcept<wbr>ion: ERROR: update or delete on<br>
&gt; table &quot;gluster_volumes&quot; violates foreign key constraint<br>
&gt; &quot;fk_storage_connection_to_glus<wbr>tervolume&quot; on table<br>
&gt; &quot;storage_server_connections&quot;<br>
&gt; Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-<wbr>bd317623ed2d) is still referenced<br>
&gt; from table &quot;storage_server_connections&quot;.<br>
&gt; Where: SQL statement &quot;DELETE<br>
&gt; FROM gluster_volumes<br>
&gt; WHERE id IN (<br>
&gt; SELECT *<br>
&gt; FROM fnSplitterUuid(v_volume_ids)<br>
&gt; )&quot;<br>
&gt; PL/pgSQL function deleteglustervolumesbyguids(ch<wbr>aracter varying) line 3 at<br>
&gt; SQL statement<br>
&gt; at<br>
&gt; org.postgresql.core.v3.QueryEx<wbr>ecutorImpl.receiveErrorRespons<wbr>e(QueryExecutorImpl.java:2157)<br>
&gt; at<br>
&gt; org.postgresql.core.v3.QueryEx<wbr>ecutorImpl.processResults(Quer<wbr>yExecutorImpl.java:1886)<br>
&gt; at<br>
&gt; org.postgresql.core.v3.QueryEx<wbr>ecutorImpl.execute(QueryExecut<wbr>orImpl.java:255)<br>
&gt; at<br>
&gt; org.postgresql.jdbc2.AbstractJ<wbr>dbc2Statement.execute(Abstract<wbr>Jdbc2Statement.java:555)<br>
&gt; at<br>
&gt; org.postgresql.jdbc2.AbstractJ<wbr>dbc2Statement.executeWithFlags<wbr>(AbstractJdbc2Statement.java:4<wbr>17)<br>
&gt; at<br>
&gt; org.postgresql.jdbc2.AbstractJ<wbr>dbc2Statement.execute(Abstract<wbr>Jdbc2Statement.java:410)<br>
&gt; at<br>
&gt; <a href="http://org.jboss.jca.adapters.jdbc.Ca" target="_blank">org.jboss.jca.adapters.jdbc.Ca</a><wbr>chedPreparedStatement.execute(<wbr>CachedPreparedStatement.java:3<wbr>03)<br>
&gt; at<br>
&gt; org.jboss.jca.adapters.jdbc.Wr<wbr>appedPreparedStatement.execute<wbr>(WrappedPreparedStatement.java<wbr>:442)<br>
&gt; at<br>
&gt; org.springframework.jdbc.core.<wbr>JdbcTemplate$6.doInCallableSta<wbr>tement(JdbcTemplate.java:1133)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at<br>
&gt; org.springframework.jdbc.core.<wbr>JdbcTemplate$6.doInCallableSta<wbr>tement(JdbcTemplate.java:1130)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; at org.springframework.jdbc.core.<wbr>JdbcTemplate.execute(JdbcTempl<wbr>ate.java:1078)<br>
&gt; [spring-jdbc.jar:4.2.4.RELEASE<wbr>]<br>
&gt; ... 24 more<br>
&gt;<br>
&gt;<br>
&gt;<br>
<br>
</div></div>This is a side effect volume deletion in the gluster side. Looks like you have storage domains created using those volumes.<br>
<div><div class="m_690619254500226376m_5030294642598411805h5"><br>
&gt; On Wed, Mar 1, 2017 at 9:49 AM, Arman Khalatyan &lt; <a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a> &gt; wrote:<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; Hi,<br>
&gt; I just tested power cut on the test system:<br>
&gt;<br>
&gt; Cluster with 3-Hosts each host has 4TB localdisk with zfs on it /zhost/01/glu<br>
&gt; folder as a brick.<br>
&gt;<br>
&gt; Glusterfs was with replicated to 3 disks with arbiter. So far so good. Vm was<br>
&gt; up an running with 5oGB OS disk: dd was showing 100-70MB/s performance with<br>
&gt; the Vm disk.<br>
&gt; I just simulated disaster powercut: with ipmi power-cycle all 3 hosts same<br>
&gt; time.<br>
&gt; the result is all hosts are green up and running but bricks are down.<br>
&gt; in the processes I can see:<br>
&gt; ps aux | grep gluster<br>
&gt; root 16156 0.8 0.0 475360 16964 ? Ssl 08:47 0:00 /usr/sbin/glusterd -p<br>
&gt; /var/run/glusterd.pid --log-level INFO<br>
&gt;<br>
&gt; What happened with my volume setup??<br>
&gt; Is it possible to recover it??<br>
&gt; [root@clei21 ~]# gluster peer status<br>
&gt; Number of Peers: 2<br>
&gt;<br>
&gt; Hostname: clei22.cls<br>
&gt; Uuid: 96b52c7e-3526-44fd-af80-14a307<wbr>3ebac2<br>
&gt; State: Peer in Cluster (Connected)<br>
&gt; Other names:<br>
&gt; 192.168.101.40<br>
&gt; 10.10.10.44<br>
&gt;<br>
&gt; Hostname: clei26.cls<br>
&gt; Uuid: c9fab907-5053-41a8-a1fa-d069f3<wbr>4e42dc<br>
&gt; State: Peer in Cluster (Connected)<br>
&gt; Other names:<br>
&gt; 10.10.10.41<br>
&gt; [root@clei21 ~]# gluster volume info<br>
&gt; No volumes present<br>
&gt; [root@clei21 ~]#<br>
<br>
</div></div>I not sure why all volumes are getting deleted after reboot. Do you see any vol files under the directory /var/lib/glusterd/vols/?. Also  /var/log/glusterfs/cmd_history<wbr>.log should have all the gluster commands executed.<br>
<br>
Regards,<br>
Ramesh<br>
<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; ______________________________<wbr>_________________<br>
&gt; Users mailing list<br>
&gt; <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
&gt; <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
&gt;<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>