<div dir="ltr">engine throws following errors:<br>2017-03-01 10:39:59,608+01 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [d7f7d83] EVENT_ID: GLUSTER_VOLUME_DELETED_FROM_CLI(4,027), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Detected deletion of volume GluReplica on cluster HaGLU, and deleted it from engine DB.<br>2017-03-01 10:39:59,610+01 ERROR [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler6) [d7f7d83] Error while removing volumes from database!: org.springframework.dao.DataIntegrityViolationException: CallableStatementCallback; SQL [{call deleteglustervolumesbyguids(?)}]; ERROR: update or delete on table "gluster_volumes" violates foreign key constraint "fk_storage_connection_to_glustervolume" on table "storage_server_connections"<br> Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still referenced from table "storage_server_connections".<br> Where: SQL statement "DELETE<br> FROM gluster_volumes<br> WHERE id IN (<br> SELECT *<br> FROM fnSplitterUuid(v_volume_ids)<br> )"<br>PL/pgSQL function deleteglustervolumesbyguids(character varying) line 3 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: update or delete on table "gluster_volumes" violates foreign key constraint "fk_storage_connection_to_glustervolume" on table "storage_server_connections"<br> Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still referenced from table "storage_server_connections".<br> Where: SQL statement "DELETE<br> FROM gluster_volumes<br> WHERE id IN (<br> SELECT *<br> FROM fnSplitterUuid(v_volume_ids)<br> )"<br>PL/pgSQL function deleteglustervolumesbyguids(character varying) line 3 at SQL statement<br> at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:243) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1094) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1130) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:405) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:365) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135) [dal.jar:]<br> at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:130) [dal.jar:]<br> at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:76) [dal.jar:]<br> at org.ovirt.engine.core.dao.gluster.GlusterVolumeDaoImpl.removeAll(GlusterVolumeDaoImpl.java:233) [dal.jar:]<br> at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.removeDeletedVolumes(GlusterSyncJob.java:521) [bll.jar:]<br> at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshVolumeData(GlusterSyncJob.java:465) [bll.jar:]<br> at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshClusterData(GlusterSyncJob.java:133) [bll.jar:]<br> at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:111) [bll.jar:]<br> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121]<br> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121]<br> at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:]<br> at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:]<br> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]<br> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_121]<br> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_121]<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]<br> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]<br>Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "gluster_volumes" violates foreign key constraint "fk_storage_connection_to_glustervolume" on table "storage_server_connections"<br> Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still referenced from table "storage_server_connections".<br> Where: SQL statement "DELETE<br> FROM gluster_volumes<br> WHERE id IN (<br> SELECT *<br> FROM fnSplitterUuid(v_volume_ids)<br> )"<br>PL/pgSQL function deleteglustervolumesbyguids(character varying) line 3 at SQL statement<br> at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)<br> at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)<br> at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)<br> at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)<br> at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)<br> at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:410)<br> at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:303)<br> at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:442)<br> at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1133) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1130) [spring-jdbc.jar:4.2.4.RELEASE]<br> at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1078) [spring-jdbc.jar:4.2.4.RELEASE]<br> ... 24 more<br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 1, 2017 at 9:49 AM, Arman Khalatyan <span dir="ltr"><<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi,<br></div>I just tested power cut on the test system:<br></div><br>Cluster with 3-Hosts each host has 4TB localdisk with zfs on it /zhost/01/glu folder as a brick.<br><br></div><div>Glusterfs was with replicated to 3 disks with arbiter. So far so good. Vm was up an running with 5oGB OS disk: dd was showing 100-70MB/s performance with the Vm disk.<br></div><div>I just simulated disaster powercut: with ipmi power-cycle all 3 hosts same time.<br></div><div>the result is all hosts are green up and running but bricks are down.<br></div><div>in the processes I can see:<br>ps aux | grep gluster <br>root 16156 0.8 0.0 475360 16964 ? Ssl 08:47 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO<br><br></div><div>What happened with my volume setup??<br></div><div>Is it possible to recover it??<br>[root@clei21 ~]# gluster peer status <br>Number of Peers: 2<br><br>Hostname: clei22.cls<br>Uuid: 96b52c7e-3526-44fd-af80-<wbr>14a3073ebac2<br>State: Peer in Cluster (Connected)<br>Other names:<br>192.168.101.40<br>10.10.10.44<br><br>Hostname: clei26.cls<br>Uuid: c9fab907-5053-41a8-a1fa-<wbr>d069f34e42dc<br>State: Peer in Cluster (Connected)<br>Other names:<br>10.10.10.41<br>[root@clei21 ~]# gluster volume info <br>No volumes present<br>[root@clei21 ~]# <br><br></div></div>
</blockquote></div><br></div>