<div dir="ltr"><div><div><div>all folders /var/lib/glusterd/vols/ are empty <br></div>In the history of one of the servers I found the command how it was created:<br><br>gluster volume create GluReplica replica 3 arbiter 1 transport TCP,RDMA 10.10.10.44:/zclei22/01/glu 10.10.10.42:/zclei21/01/glu 10.10.10.41:/zclei26/01/glu<br><br></div>But executing this command it claims that:<br>volume create: GluReplica: failed: /zclei22/01/glu is already part of a volume<br><br></div>Any chance to force it?<br><br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 1, 2017 at 12:13 PM, Ramesh Nachimuthu <span dir="ltr"><<a href="mailto:rnachimu@redhat.com" target="_blank">rnachimu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
----- Original Message -----<br>
> From: "Arman Khalatyan" <<a href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a>><br>
> To: "users" <<a href="mailto:users@ovirt.org">users@ovirt.org</a>><br>
> Sent: Wednesday, March 1, 2017 3:10:38 PM<br>
> Subject: Re: [ovirt-users] Gluster setup disappears any chance to recover?<br>
><br>
> engine throws following errors:<br>
> 2017-03-01 10:39:59,608+01 WARN<br>
> [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]<br>
> (DefaultQuartzScheduler6) [d7f7d83] EVENT_ID:<br>
> GLUSTER_VOLUME_DELETED_FROM_<wbr>CLI(4,027), Correlation ID: null, Call Stack:<br>
> null, Custom Event ID: -1, Message: Detected deletion of volume GluReplica<br>
> on cluster HaGLU, and deleted it from engine DB.<br>
> 2017-03-01 10:39:59,610+01 ERROR<br>
> [org.ovirt.engine.core.bll.<wbr>gluster.GlusterSyncJob] (DefaultQuartzScheduler6)<br>
> [d7f7d83] Error while removing volumes from database!:<br>
> org.springframework.dao.<wbr>DataIntegrityViolationExceptio<wbr>n:<br>
> CallableStatementCallback; SQL [{call deleteglustervolumesbyguids(?)<wbr>}];<br>
> ERROR: update or delete on table "gluster_volumes" violates foreign key<br>
> constraint "fk_storage_connection_to_<wbr>glustervolume" on table<br>
> "storage_server_connections"<br>
> Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-<wbr>bd317623ed2d) is still referenced<br>
> from table "storage_server_connections".<br>
> Where: SQL statement "DELETE<br>
> FROM gluster_volumes<br>
> WHERE id IN (<br>
> SELECT *<br>
> FROM fnSplitterUuid(v_volume_ids)<br>
> )"<br>
> PL/pgSQL function deleteglustervolumesbyguids(<wbr>character varying) line 3 at<br>
> SQL statement; nested exception is org.postgresql.util.<wbr>PSQLException: ERROR:<br>
> update or delete on table "gluster_volumes" violates foreign key constraint<br>
> "fk_storage_connection_to_<wbr>glustervolume" on table<br>
> "storage_server_connections"<br>
> Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-<wbr>bd317623ed2d) is still referenced<br>
> from table "storage_server_connections".<br>
> Where: SQL statement "DELETE<br>
> FROM gluster_volumes<br>
> WHERE id IN (<br>
> SELECT *<br>
> FROM fnSplitterUuid(v_volume_ids)<br>
> )"<br>
> PL/pgSQL function deleteglustervolumesbyguids(<wbr>character varying) line 3 at<br>
> SQL statement<br>
> at<br>
> org.springframework.jdbc.<wbr>support.<wbr>SQLErrorCodeSQLExceptionTransl<wbr>ator.doTranslate(<wbr>SQLErrorCodeSQLExceptionTransl<wbr>ator.java:243)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at<br>
> org.springframework.jdbc.<wbr>support.<wbr>AbstractFallbackSQLExceptionTr<wbr>anslator.translate(<wbr>AbstractFallbackSQLExceptionTr<wbr>anslator.java:73)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at org.springframework.jdbc.core.<wbr>JdbcTemplate.execute(<wbr>JdbcTemplate.java:1094)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at org.springframework.jdbc.core.<wbr>JdbcTemplate.call(<wbr>JdbcTemplate.java:1130)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at<br>
> org.springframework.jdbc.core.<wbr>simple.AbstractJdbcCall.<wbr>executeCallInternal(<wbr>AbstractJdbcCall.java:405)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at<br>
> org.springframework.jdbc.core.<wbr>simple.AbstractJdbcCall.<wbr>doExecute(AbstractJdbcCall.<wbr>java:365)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at<br>
> org.springframework.jdbc.core.<wbr>simple.SimpleJdbcCall.execute(<wbr>SimpleJdbcCall.java:198)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at<br>
> org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>SimpleJdbcCallsHandler.<wbr>executeImpl(<wbr>SimpleJdbcCallsHandler.java:<wbr>135)<br>
> [dal.jar:]<br>
> at<br>
> org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>SimpleJdbcCallsHandler.<wbr>executeImpl(<wbr>SimpleJdbcCallsHandler.java:<wbr>130)<br>
> [dal.jar:]<br>
> at<br>
> org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>SimpleJdbcCallsHandler.<wbr>executeModification(<wbr>SimpleJdbcCallsHandler.java:<wbr>76)<br>
> [dal.jar:]<br>
> at<br>
> org.ovirt.engine.core.dao.<wbr>gluster.GlusterVolumeDaoImpl.<wbr>removeAll(<wbr>GlusterVolumeDaoImpl.java:233)<br>
> [dal.jar:]<br>
> at<br>
> org.ovirt.engine.core.bll.<wbr>gluster.GlusterSyncJob.<wbr>removeDeletedVolumes(<wbr>GlusterSyncJob.java:521)<br>
> [bll.jar:]<br>
> at<br>
> org.ovirt.engine.core.bll.<wbr>gluster.GlusterSyncJob.<wbr>refreshVolumeData(<wbr>GlusterSyncJob.java:465)<br>
> [bll.jar:]<br>
> at<br>
> org.ovirt.engine.core.bll.<wbr>gluster.GlusterSyncJob.<wbr>refreshClusterData(<wbr>GlusterSyncJob.java:133)<br>
> [bll.jar:]<br>
> at<br>
> org.ovirt.engine.core.bll.<wbr>gluster.GlusterSyncJob.<wbr>refreshLightWeightData(<wbr>GlusterSyncJob.java:111)<br>
> [bll.jar:]<br>
> at sun.reflect.<wbr>NativeMethodAccessorImpl.<wbr>invoke0(Native Method)<br>
> [rt.jar:1.8.0_121]<br>
> at<br>
> sun.reflect.<wbr>NativeMethodAccessorImpl.<wbr>invoke(<wbr>NativeMethodAccessorImpl.java:<wbr>62)<br>
> [rt.jar:1.8.0_121]<br>
> at<br>
> sun.reflect.<wbr>DelegatingMethodAccessorImpl.<wbr>invoke(<wbr>DelegatingMethodAccessorImpl.<wbr>java:43)<br>
> [rt.jar:1.8.0_121]<br>
> at java.lang.reflect.Method.<wbr>invoke(Method.java:498) [rt.jar:1.8.0_121]<br>
> at<br>
> org.ovirt.engine.core.utils.<wbr>timer.JobWrapper.invokeMethod(<wbr>JobWrapper.java:77)<br>
> [scheduler.jar:]<br>
> at org.ovirt.engine.core.utils.<wbr>timer.JobWrapper.execute(<wbr>JobWrapper.java:51)<br>
> [scheduler.jar:]<br>
> at org.quartz.core.JobRunShell.<wbr>run(JobRunShell.java:213) [quartz.jar:]<br>
> at java.util.concurrent.<wbr>Executors$RunnableAdapter.<wbr>call(Executors.java:511)<br>
> [rt.jar:1.8.0_121]<br>
> at java.util.concurrent.<wbr>FutureTask.run(FutureTask.<wbr>java:266)<br>
> [rt.jar:1.8.0_121]<br>
> at<br>
> java.util.concurrent.<wbr>ThreadPoolExecutor.runWorker(<wbr>ThreadPoolExecutor.java:1142)<br>
> [rt.jar:1.8.0_121]<br>
> at<br>
> java.util.concurrent.<wbr>ThreadPoolExecutor$Worker.run(<wbr>ThreadPoolExecutor.java:617)<br>
> [rt.jar:1.8.0_121]<br>
> at java.lang.Thread.run(Thread.<wbr>java:745) [rt.jar:1.8.0_121]<br>
> Caused by: org.postgresql.util.<wbr>PSQLException: ERROR: update or delete on<br>
> table "gluster_volumes" violates foreign key constraint<br>
> "fk_storage_connection_to_<wbr>glustervolume" on table<br>
> "storage_server_connections"<br>
> Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-<wbr>bd317623ed2d) is still referenced<br>
> from table "storage_server_connections".<br>
> Where: SQL statement "DELETE<br>
> FROM gluster_volumes<br>
> WHERE id IN (<br>
> SELECT *<br>
> FROM fnSplitterUuid(v_volume_ids)<br>
> )"<br>
> PL/pgSQL function deleteglustervolumesbyguids(<wbr>character varying) line 3 at<br>
> SQL statement<br>
> at<br>
> org.postgresql.core.v3.<wbr>QueryExecutorImpl.<wbr>receiveErrorResponse(<wbr>QueryExecutorImpl.java:2157)<br>
> at<br>
> org.postgresql.core.v3.<wbr>QueryExecutorImpl.<wbr>processResults(<wbr>QueryExecutorImpl.java:1886)<br>
> at<br>
> org.postgresql.core.v3.<wbr>QueryExecutorImpl.execute(<wbr>QueryExecutorImpl.java:255)<br>
> at<br>
> org.postgresql.jdbc2.<wbr>AbstractJdbc2Statement.<wbr>execute(<wbr>AbstractJdbc2Statement.java:<wbr>555)<br>
> at<br>
> org.postgresql.jdbc2.<wbr>AbstractJdbc2Statement.<wbr>executeWithFlags(<wbr>AbstractJdbc2Statement.java:<wbr>417)<br>
> at<br>
> org.postgresql.jdbc2.<wbr>AbstractJdbc2Statement.<wbr>execute(<wbr>AbstractJdbc2Statement.java:<wbr>410)<br>
> at<br>
> org.jboss.jca.adapters.jdbc.<wbr>CachedPreparedStatement.<wbr>execute(<wbr>CachedPreparedStatement.java:<wbr>303)<br>
> at<br>
> org.jboss.jca.adapters.jdbc.<wbr>WrappedPreparedStatement.<wbr>execute(<wbr>WrappedPreparedStatement.java:<wbr>442)<br>
> at<br>
> org.springframework.jdbc.core.<wbr>JdbcTemplate$6.<wbr>doInCallableStatement(<wbr>JdbcTemplate.java:1133)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at<br>
> org.springframework.jdbc.core.<wbr>JdbcTemplate$6.<wbr>doInCallableStatement(<wbr>JdbcTemplate.java:1130)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> at org.springframework.jdbc.core.<wbr>JdbcTemplate.execute(<wbr>JdbcTemplate.java:1078)<br>
> [spring-jdbc.jar:4.2.4.<wbr>RELEASE]<br>
> ... 24 more<br>
><br>
><br>
><br>
<br>
</div></div>This is a side effect volume deletion in the gluster side. Looks like you have storage domains created using those volumes.<br>
<div><div class="h5"><br>
> On Wed, Mar 1, 2017 at 9:49 AM, Arman Khalatyan < <a href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a> > wrote:<br>
><br>
><br>
><br>
> Hi,<br>
> I just tested power cut on the test system:<br>
><br>
> Cluster with 3-Hosts each host has 4TB localdisk with zfs on it /zhost/01/glu<br>
> folder as a brick.<br>
><br>
> Glusterfs was with replicated to 3 disks with arbiter. So far so good. Vm was<br>
> up an running with 5oGB OS disk: dd was showing 100-70MB/s performance with<br>
> the Vm disk.<br>
> I just simulated disaster powercut: with ipmi power-cycle all 3 hosts same<br>
> time.<br>
> the result is all hosts are green up and running but bricks are down.<br>
> in the processes I can see:<br>
> ps aux | grep gluster<br>
> root 16156 0.8 0.0 475360 16964 ? Ssl 08:47 0:00 /usr/sbin/glusterd -p<br>
> /var/run/glusterd.pid --log-level INFO<br>
><br>
> What happened with my volume setup??<br>
> Is it possible to recover it??<br>
> [root@clei21 ~]# gluster peer status<br>
> Number of Peers: 2<br>
><br>
> Hostname: clei22.cls<br>
> Uuid: 96b52c7e-3526-44fd-af80-<wbr>14a3073ebac2<br>
> State: Peer in Cluster (Connected)<br>
> Other names:<br>
> 192.168.101.40<br>
> 10.10.10.44<br>
><br>
> Hostname: clei26.cls<br>
> Uuid: c9fab907-5053-41a8-a1fa-<wbr>d069f34e42dc<br>
> State: Peer in Cluster (Connected)<br>
> Other names:<br>
> 10.10.10.41<br>
> [root@clei21 ~]# gluster volume info<br>
> No volumes present<br>
> [root@clei21 ~]#<br>
<br>
</div></div>I not sure why all volumes are getting deleted after reboot. Do you see any vol files under the directory /var/lib/glusterd/vols/?. Also /var/log/glusterfs/cmd_<wbr>history.log should have all the gluster commands executed.<br>
<br>
Regards,<br>
Ramesh<br>
<br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
><br>
</blockquote></div><br></div>