Re: [ovirt-users] Gluster setup disappears any chance to recover?

From: "Arman Khalatyan" <arm2arm(a)gmail.com&gt; To: "Ramesh Nachimuthu" <rnachimu(a)redhat.com&gt; Cc: "users" <users(a)ovirt.org&gt;, "Sahina Bose" <sabose(a)redhat.com&gt; Sent: Wednesday, March 1, 2017 11:22:32 PM Subject: Re: [ovirt-users] Gluster setup disappears any chance to recover? ok I will answer by my self: yes gluster daemon is managed by vdms:) and to recover lost config simply one should add "force" keyword gluster volume create GluReplica replica 3 arbiter 1 transport TCP,RDMA 10.10.10.44:/zclei22/01/glu 10.10.10.42:/zclei21/01/glu 10.10.10.41:/zclei26/01/glu force now everything is up an running ! one annoying thing is epel dependency in the zfs and conflicting ovirt... every time one need to enable and then disable epel.

On Wed, Mar 1, 2017 at 5:33 PM, Arman Khalatyan <arm2arm(a)gmail.com&gt; wrote: > ok Finally by single brick up and running so I can access to data. > Now the question is do we need to run glusterd daemon on startup? or it is > managed by vdsmd? > > > On Wed, Mar 1, 2017 at 2:36 PM, Arman Khalatyan <arm2arm(a)gmail.com&gt; wrote: > >> all folders /var/lib/glusterd/vols/ are empty >> In the history of one of the servers I found the command how it was >> created: >> >> gluster volume create GluReplica replica 3 arbiter 1 transport TCP,RDMA >> 10.10.10.44:/zclei22/01/glu 10.10.10.42:/zclei21/01/glu 10.10.10.41: >> /zclei26/01/glu >> >> But executing this command it claims that: >> volume create: GluReplica: failed: /zclei22/01/glu is already part of a >> volume >> >> Any chance to force it? >> >> >> >> On Wed, Mar 1, 2017 at 12:13 PM, Ramesh Nachimuthu <rnachimu(a)redhat.com&gt; >> wrote: >> >>> >>> >>> >>> >>> ----- Original Message ----- >>> > From: "Arman Khalatyan" <arm2arm(a)gmail.com&gt; >>> > To: "users" <users(a)ovirt.org&gt; >>> > Sent: Wednesday, March 1, 2017 3:10:38 PM >>> > Subject: Re: [ovirt-users] Gluster setup disappears any chance to >>> recover? >>> > >>> > engine throws following errors: >>> > 2017-03-01 10:39:59,608+01 WARN >>> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> > (DefaultQuartzScheduler6) [d7f7d83] EVENT_ID: >>> > GLUSTER_VOLUME_DELETED_FROM_CLI(4,027), Correlation ID: null, Call >>> Stack: >>> > null, Custom Event ID: -1, Message: Detected deletion of volume >>> GluReplica >>> > on cluster HaGLU, and deleted it from engine DB. >>> > 2017-03-01 10:39:59,610+01 ERROR >>> > [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] >>> (DefaultQuartzScheduler6) >>> > [d7f7d83] Error while removing volumes from database!: >>> > org.springframework.dao.DataIntegrityViolationException: >>> > CallableStatementCallback; SQL [{call deleteglustervolumesbyguids(?) >>> }]; >>> > ERROR: update or delete on table "gluster_volumes" violates foreign key >>> > constraint "fk_storage_connection_to_glustervolume" on table >>> > "storage_server_connections" >>> > Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still >>> referenced >>> > from table "storage_server_connections". >>> > Where: SQL statement "DELETE >>> > FROM gluster_volumes >>> > WHERE id IN ( >>> > SELECT * >>> > FROM fnSplitterUuid(v_volume_ids) >>> > )" >>> > PL/pgSQL function deleteglustervolumesbyguids(character varying) line >>> 3 at >>> > SQL statement; nested exception is org.postgresql.util.PSQLException: >>> ERROR: >>> > update or delete on table "gluster_volumes" violates foreign key >>> constraint >>> > "fk_storage_connection_to_glustervolume" on table >>> > "storage_server_connections" >>> > Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still >>> referenced >>> > from table "storage_server_connections". >>> > Where: SQL statement "DELETE >>> > FROM gluster_volumes >>> > WHERE id IN ( >>> > SELECT * >>> > FROM fnSplitterUuid(v_volume_ids) >>> > )" >>> > PL/pgSQL function deleteglustervolumesbyguids(character varying) line >>> 3 at >>> > SQL statement >>> > at >>> > org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTra >>> nslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:243) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at >>> > org.springframework.jdbc.support.AbstractFallbackSQLExceptio >>> nTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTempl >>> ate.java:1094) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate >>> .java:1130) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at >>> > org.springframework.jdbc.core.simple.AbstractJdbcCall.execut >>> eCallInternal(AbstractJdbcCall.java:405) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at >>> > org.springframework.jdbc.core.simple.AbstractJdbcCall.doExec >>> ute(AbstractJdbcCall.java:365) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at >>> > org.springframework.jdbc.core.simple.SimpleJdbcCall.execute( >>> SimpleJdbcCall.java:198) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at >>> > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.ex >>> ecuteImpl(SimpleJdbcCallsHandler.java:135) >>> > [dal.jar:] >>> > at >>> > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.ex >>> ecuteImpl(SimpleJdbcCallsHandler.java:130) >>> > [dal.jar:] >>> > at >>> > org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.ex >>> ecuteModification(SimpleJdbcCallsHandler.java:76) >>> > [dal.jar:] >>> > at >>> > org.ovirt.engine.core.dao.gluster.GlusterVolumeDaoImpl.remov >>> eAll(GlusterVolumeDaoImpl.java:233) >>> > [dal.jar:] >>> > at >>> > org.ovirt.engine.core.bll.gluster.GlusterSyncJob.removeDelet >>> edVolumes(GlusterSyncJob.java:521) >>> > [bll.jar:] >>> > at >>> > org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshVolu >>> meData(GlusterSyncJob.java:465) >>> > [bll.jar:] >>> > at >>> > org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshClus >>> terData(GlusterSyncJob.java:133) >>> > [bll.jar:] >>> > at >>> > org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLigh >>> tWeightData(GlusterSyncJob.java:111) >>> > [bll.jar:] >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> > [rt.jar:1.8.0_121] >>> > at >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:62) >>> > [rt.jar:1.8.0_121] >>> > at >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> > [rt.jar:1.8.0_121] >>> > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] >>> > at >>> > org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(Jo >>> bWrapper.java:77) >>> > [scheduler.jar:] >>> > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrap >>> per.java:51) >>> > [scheduler.jar:] >>> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] >>> > at java.util.concurrent.Executors$RunnableAdapter.call(Executor >>> s.java:511) >>> > [rt.jar:1.8.0_121] >>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> > [rt.jar:1.8.0_121] >>> > at >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> > [rt.jar:1.8.0_121] >>> > at >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> > [rt.jar:1.8.0_121] >>> > at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121] >>> > Caused by: org.postgresql.util.PSQLException: ERROR: update or delete >>> on >>> > table "gluster_volumes" violates foreign key constraint >>> > "fk_storage_connection_to_glustervolume" on table >>> > "storage_server_connections" >>> > Detail: Key (id)=(3d8bfa9d-1c83-46ac-b4e9-bd317623ed2d) is still >>> referenced >>> > from table "storage_server_connections". >>> > Where: SQL statement "DELETE >>> > FROM gluster_volumes >>> > WHERE id IN ( >>> > SELECT * >>> > FROM fnSplitterUuid(v_volume_ids) >>> > )" >>> > PL/pgSQL function deleteglustervolumesbyguids(character varying) line >>> 3 at >>> > SQL statement >>> > at >>> > org.postgresql.core.v3.QueryExecutorImpl.receiveErrorRespons >>> e(QueryExecutorImpl.java:2157) >>> > at >>> > org.postgresql.core.v3.QueryExecutorImpl.processResults(Quer >>> yExecutorImpl.java:1886) >>> > at >>> > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecut >>> orImpl.java:255) >>> > at >>> > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(Abstract >>> Jdbc2Statement.java:555) >>> > at >>> > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags >>> (AbstractJdbc2Statement.java:417) >>> > at >>> > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(Abstract >>> Jdbc2Statement.java:410) >>> > at >>> > org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute( >>> CachedPreparedStatement.java:303) >>> > at >>> > org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute >>> (WrappedPreparedStatement.java:442) >>> > at >>> > org.springframework.jdbc.core.JdbcTemplate$6.doInCallableSta >>> tement(JdbcTemplate.java:1133) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at >>> > org.springframework.jdbc.core.JdbcTemplate$6.doInCallableSta >>> tement(JdbcTemplate.java:1130) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTempl >>> ate.java:1078) >>> > [spring-jdbc.jar:4.2.4.RELEASE] >>> > ... 24 more >>> > >>> > >>> > >>> >>> This is a side effect volume deletion in the gluster side. Looks like >>> you have storage domains created using those volumes. >>> >>> > On Wed, Mar 1, 2017 at 9:49 AM, Arman Khalatyan < arm2arm(a)gmail.com > >>> wrote: >>> > >>> > >>> > >>> > Hi, >>> > I just tested power cut on the test system: >>> > >>> > Cluster with 3-Hosts each host has 4TB localdisk with zfs on it >>> /zhost/01/glu >>> > folder as a brick. >>> > >>> > Glusterfs was with replicated to 3 disks with arbiter. So far so good. >>> Vm was >>> > up an running with 5oGB OS disk: dd was showing 100-70MB/s performance >>> with >>> > the Vm disk. >>> > I just simulated disaster powercut: with ipmi power-cycle all 3 hosts >>> same >>> > time. >>> > the result is all hosts are green up and running but bricks are down. >>> > in the processes I can see: >>> > ps aux | grep gluster >>> > root 16156 0.8 0.0 475360 16964 ? Ssl 08:47 0:00 /usr/sbin/glusterd -p >>> > /var/run/glusterd.pid --log-level INFO >>> > >>> > What happened with my volume setup?? >>> > Is it possible to recover it?? >>> > [root@clei21 ~]# gluster peer status >>> > Number of Peers: 2 >>> > >>> > Hostname: clei22.cls >>> > Uuid: 96b52c7e-3526-44fd-af80-14a3073ebac2 >>> > State: Peer in Cluster (Connected) >>> > Other names: >>> > 192.168.101.40 >>> > 10.10.10.44 >>> > >>> > Hostname: clei26.cls >>> > Uuid: c9fab907-5053-41a8-a1fa-d069f34e42dc >>> > State: Peer in Cluster (Connected) >>> > Other names: >>> > 10.10.10.41 >>> > [root@clei21 ~]# gluster volume info >>> > No volumes present >>> > [root@clei21 ~]# >>> >>> I not sure why all volumes are getting deleted after reboot. Do you see >>> any vol files under the directory /var/lib/glusterd/vols/?. Also >>> /var/log/glusterfs/cmd_history.log should have all the gluster commands >>> executed. >>> >>> Regards, >>> Ramesh >>> >>> > >>> > >>> > >>> > _______________________________________________ >>> > Users mailing list >>> > Users(a)ovirt.org >>> > http://lists.ovirt.org/mailman/listinfo/users >>> > >>> >> >> >