[ovirt-users] Gluster setup disappears any chance to recover?

Arman Khalatyan arm2arm at gmail.com
Wed Mar 1 09:40:38 UTC 2017 

engine throws following errors:
2017-03-01 10:39:59,608+01 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler6) [d7f7d83] EVENT_ID:
GLUSTER_VOLUME_DELETED_FROM_CLI(4,027), Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: Detected deletion of volume GluReplica
on cluster HaGLU, and deleted it from engine DB.
2017-03-01 10:39:59,610+01 ERROR
[org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler6) [d7f7d83] Error while removing volumes from
database!: org.springframework.dao.DataIntegrityViolationException:
CallableStatementCallback; SQL [{call deleteglustervolumesbyguids(?)}];
ERROR: update or delete on table "gluster_volumes" violates foreign key
constraint "fk_storage_connection_to_glustervolume" on table
"storage_server_connections"
  Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still
referenced from table "storage_server_connections".
  Where: SQL statement "DELETE
    FROM gluster_volumes
    WHERE id IN (
            SELECT *
            FROM fnSplitterUuid(v_volume_ids)
            )"
PL/pgSQL function deleteglustervolumesbyguids(character varying) line 3 at
SQL statement; nested exception is org.postgresql.util.PSQLException:
ERROR: update or delete on table "gluster_volumes" violates foreign key
constraint "fk_storage_connection_to_glustervolume" on table
"storage_server_connections"
  Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still
referenced from table "storage_server_connections".
  Where: SQL statement "DELETE
    FROM gluster_volumes
    WHERE id IN (
            SELECT *
            FROM fnSplitterUuid(v_volume_ids)
            )"
PL/pgSQL function deleteglustervolumesbyguids(character varying) line 3 at
SQL statement
    at
org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:243)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1094)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1130)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:405)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:365)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135)
[dal.jar:]
    at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:130)
[dal.jar:]
    at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:76)
[dal.jar:]
    at
org.ovirt.engine.core.dao.gluster.GlusterVolumeDaoImpl.removeAll(GlusterVolumeDaoImpl.java:233)
[dal.jar:]
    at
org.ovirt.engine.core.bll.gluster.GlusterSyncJob.removeDeletedVolumes(GlusterSyncJob.java:521)
[bll.jar:]
    at
org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshVolumeData(GlusterSyncJob.java:465)
[bll.jar:]
    at
org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshClusterData(GlusterSyncJob.java:133)
[bll.jar:]
    at
org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:111)
[bll.jar:]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_121]
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_121]
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_121]
    at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121]
    at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77)
[scheduler.jar:]
    at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51)
[scheduler.jar:]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_121]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_121]
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_121]
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_121]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on
table "gluster_volumes" violates foreign key constraint
"fk_storage_connection_to_glustervolume" on table
"storage_server_connections"
  Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still
referenced from table "storage_server_connections".
  Where: SQL statement "DELETE
    FROM gluster_volumes
    WHERE id IN (
            SELECT *
            FROM fnSplitterUuid(v_volume_ids)
            )"
PL/pgSQL function deleteglustervolumesbyguids(character varying) line 3 at
SQL statement
    at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
    at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
    at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
    at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
    at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
    at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:410)
    at
org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:303)
    at
org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:442)
    at
org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1133)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1130)
[spring-jdbc.jar:4.2.4.RELEASE]
    at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1078)
[spring-jdbc.jar:4.2.4.RELEASE]
    ... 24 more



On Wed, Mar 1, 2017 at 9:49 AM, Arman Khalatyan <arm2arm at gmail.com> wrote:

> Hi,
> I just tested power cut on the test system:
>
> Cluster with 3-Hosts each host has 4TB localdisk with zfs on it
> /zhost/01/glu folder as a brick.
>
> Glusterfs was with replicated to 3 disks with arbiter. So far so good. Vm
> was up an running with 5oGB OS disk: dd was showing 100-70MB/s performance
> with the Vm disk.
> I just simulated disaster powercut: with ipmi power-cycle all 3 hosts same
> time.
> the result is all hosts are green up and running but bricks are down.
> in the processes I can see:
> ps aux | grep gluster
> root     16156  0.8  0.0 475360 16964 ?        Ssl  08:47   0:00
> /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
>
> What happened with my volume setup??
> Is it possible to recover it??
> [root at clei21 ~]# gluster peer status
> Number of Peers: 2
>
> Hostname: clei22.cls
> Uuid: 96b52c7e-3526-44fd-af80-14a3073ebac2
> State: Peer in Cluster (Connected)
> Other names:
> 192.168.101.40
> 10.10.10.44
>
> Hostname: clei26.cls
> Uuid: c9fab907-5053-41a8-a1fa-d069f34e42dc
> State: Peer in Cluster (Connected)
> Other names:
> 10.10.10.41
> [root at clei21 ~]# gluster volume info
> No volumes present
> [root at clei21 ~]#
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170301/3ba79f7b/attachment-0001.html>

More information about the Users mailing list