Re: [ovirt-users] Gluster command [<UNKNOWN>] failed on server...

24 Nov 2014

      This is a multi-part message in MIME format.
--------------080701090102060806020404
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

Can you send the corresponding error in vdsm.log from the host?

Also check if glusterd service is running.

Thanks,
Kanagaraj

On 11/24/2014 03:39 PM, Punit Dambiwal wrote:
...
Hi,
After reboot my Hypervisior host can not activate again in the cluster 
and failed with the following error :-
Gluster command [<UNKNOWN>] failed on server...
Engine logs :-
2014-11-24 18:05:28,397 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) 
START, GlusterVolumesListVDSCommand(HostName = Compute4, HostId = 
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90
2014-11-24 18:05:30,609 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) 
FINISH, GlusterVolumesListVDSCommand, return: 
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d95203e0}, 
log id: 5f251c90
2014-11-24 18:05:33,768 INFO 
 [org.ovirt.engine.core.bll.ActivateVdsCommand] 
(ajp--127.0.0.1-8702-8) [287d570d] Lock Acquired to object EngineLock 
[exclusiveLocks= key: 0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
, sharedLocks= ]
2014-11-24 18:05:33,795 INFO 
 [org.ovirt.engine.core.bll.ActivateVdsCommand] 
(org.ovirt.thread.pool-8-thread-45) [287d570d] Running command: 
ActivateVdsCommand internal: false. Entities affected :  ID: 
0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDSAction group 
MANIPULATE_HOST with role type ADMIN
2014-11-24 18:05:33,796 INFO 
 [org.ovirt.engine.core.bll.ActivateVdsCommand] 
(org.ovirt.thread.pool-8-thread-45) [287d570d] Before acquiring lock 
in order to prevent monitoring for host Compute5 from data-center SV_WTC
2014-11-24 18:05:33,797 INFO 
 [org.ovirt.engine.core.bll.ActivateVdsCommand] 
(org.ovirt.thread.pool-8-thread-45) [287d570d] Lock acquired, from now 
a monitoring of host will be skipped for host Compute5 from 
data-center SV_WTC
2014-11-24 18:05:33,817 INFO 
 [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] 
(org.ovirt.thread.pool-8-thread-45) [287d570d] START, 
SetVdsStatusVDSCommand(HostName = Compute5, HostId = 
0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned, 
nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1cbc7311
2014-11-24 18:05:33,820 INFO 
 [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] 
(org.ovirt.thread.pool-8-thread-45) [287d570d] FINISH, 
SetVdsStatusVDSCommand, log id: 1cbc7311
2014-11-24 18:05:34,086 INFO 
 [org.ovirt.engine.core.bll.ActivateVdsCommand] 
(org.ovirt.thread.pool-8-thread-45) Activate finished. Lock released. 
Monitoring can run now for host Compute5 from data-center SV_WTC
2014-11-24 18:05:34,088 INFO 
 [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-45) 
Correlation ID: 287d570d, Job ID: 
5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call Stack: null, Custom Event 
ID: -1, Message: Host Compute5 was activated by admin.
2014-11-24 18:05:34,090 INFO 
 [org.ovirt.engine.core.bll.ActivateVdsCommand] 
(org.ovirt.thread.pool-8-thread-45) Lock freed to object EngineLock 
[exclusiveLocks= key: 0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
, sharedLocks= ]
2014-11-24 18:05:35,792 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) 
[3706e836] START, GlusterVolumesListVDSCommand(HostName = Compute4, 
HostId = 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832
2014-11-24 18:05:37,064 INFO 
 [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] 
(DefaultQuartzScheduler_Worker-69) START, 
GetHardwareInfoVDSCommand(HostName = Compute5, HostId = 
0bf6b00f-7947-4411-b55a-cc5eea2b381a, 
vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log id: 6d560cc2
2014-11-24 18:05:37,074 INFO 
 [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] 
(DefaultQuartzScheduler_Worker-69) FINISH, GetHardwareInfoVDSCommand, 
log id: 6d560cc2
2014-11-24 18:05:37,093 WARN 
 [org.ovirt.engine.core.vdsbroker.VdsManager] 
(DefaultQuartzScheduler_Worker-69) Host Compute5 is running with 
disabled SELinux.
2014-11-24 18:05:37,127 INFO 
 [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] 
(DefaultQuartzScheduler_Worker-69) [2b4a51cf] Running command: 
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities 
affected :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
2014-11-24 18:05:37,147 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler_Worker-69) 
[2b4a51cf] START, GlusterServersListVDSCommand(HostName = Compute5, 
HostId = 0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87
2014-11-24 18:05:37,164 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler_Worker-69) 
[2b4a51cf] FINISH, GlusterServersListVDSCommand, log id: 4faed87
2014-11-24 18:05:37,189 INFO 
 [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] 
(DefaultQuartzScheduler_Worker-69) [4a84c4e5] Running command: 
SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 
0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
2014-11-24 18:05:37,206 INFO 
 [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] 
(DefaultQuartzScheduler_Worker-69) [4a84c4e5] START, 
SetVdsStatusVDSCommand(HostName = Compute5, HostId = 
0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=NonOperational, 
nonOperationalReason=GLUSTER_COMMAND_FAILED, 
stopSpmFailureLogged=false), log id: fed5617
2014-11-24 18:05:37,209 INFO 
 [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] 
(DefaultQuartzScheduler_Worker-69) [4a84c4e5] FINISH, 
SetVdsStatusVDSCommand, log id: fed5617
2014-11-24 18:05:37,223 ERROR 
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation ID: 
4a84c4e5, Job ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b, Call Stack: 
null, Custom Event ID: -1, Message: Gluster command [<UNKNOWN>] failed 
on server Compute5.
2014-11-24 18:05:37,243 INFO 
 [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) 
[4a84c4e5] Correlation ID: null, Call Stack: null, Custom Event ID: 
-1, Message: Status of host Compute5 was set to NonOperational.
2014-11-24 18:05:37,272 INFO 
 [org.ovirt.engine.core.bll.HandleVdsVersionCommand] 
(DefaultQuartzScheduler_Worker-69) [a0c8a7f] Running command: 
HandleVdsVersionCommand internal: true. Entities affected :  ID: 
0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
2014-11-24 18:05:37,274 INFO 
 [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] 
(DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host 
0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5 is already in 
NonOperational status for reason GLUSTER_COMMAND_FAILED. 
SetNonOperationalVds command is skipped.
2014-11-24 18:05:38,065 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) 
[3706e836] FINISH, GlusterVolumesListVDSCommand, return: 
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4e72a1b1}, 
log id: 48a0c832
2014-11-24 18:05:43,243 INFO 
 [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-35) 
START, GlusterVolumesListVDSCommand(HostName = Compute4, HostId = 
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc
^C
[root@ccr01 ~]#
Thanks,
Punit
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------080701090102060806020404
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Can you send the corresponding error in vdsm.log from the host?<br>
    <br>
    Also check if glusterd service is running.<br>
    <br>
    Thanks,<br>
    Kanagaraj<br>
    <br>
    <div class="moz-cite-prefix">On 11/24/2014 03:39 PM, Punit Dambiwal
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAGZcrB=OPQ2ikZdEwGCeMbDDdPE6rJ68n6C0XuZAadhsX2eH+w@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>Hi,</div>
        <div><br>
        </div>
        <div>After reboot my Hypervisior host can not activate again in
          the cluster and failed with the following error :- </div>
        <div><br>
        </div>
        <div>Gluster command [<UNKNOWN>] failed on server...<br>
        </div>
        <div><br>
        </div>
        <div>Engine logs :- </div>
        <div><br>
        </div>
        <div>2014-11-24 18:05:28,397 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
          (DefaultQuartzScheduler_Worker-64) START,
          GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
          33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90</div>
        <div>2014-11-24 18:05:30,609 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
          (DefaultQuartzScheduler_Worker-64) FINISH,
          GlusterVolumesListVDSCommand, return:
          {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d95203e0},
          log id: 5f251c90</div>
        <div>2014-11-24 18:05:33,768 INFO
           [org.ovirt.engine.core.bll.ActivateVdsCommand]
          (ajp--127.0.0.1-8702-8) [287d570d] Lock Acquired to object
          EngineLock [exclusiveLocks= key:
          0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS</div>
        <div>, sharedLocks= ]</div>
        <div>2014-11-24 18:05:33,795 INFO
           [org.ovirt.engine.core.bll.ActivateVdsCommand]
          (org.ovirt.thread.pool-8-thread-45) [287d570d] Running
          command: ActivateVdsCommand internal: false. Entities affected
          :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDSAction
          group MANIPULATE_HOST with role type ADMIN</div>
        <div>2014-11-24 18:05:33,796 INFO
           [org.ovirt.engine.core.bll.ActivateVdsCommand]
          (org.ovirt.thread.pool-8-thread-45) [287d570d] Before
          acquiring lock in order to prevent monitoring for host
          Compute5 from data-center SV_WTC</div>
        <div>2014-11-24 18:05:33,797 INFO
           [org.ovirt.engine.core.bll.ActivateVdsCommand]
          (org.ovirt.thread.pool-8-thread-45) [287d570d] Lock acquired,
          from now a monitoring of host will be skipped for host
          Compute5 from data-center SV_WTC</div>
        <div>2014-11-24 18:05:33,817 INFO
           [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
          (org.ovirt.thread.pool-8-thread-45) [287d570d] START,
          SetVdsStatusVDSCommand(HostName = Compute5, HostId =
          0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned,
          nonOperationalReason=NONE, stopSpmFailureLogged=false), log
          id: 1cbc7311</div>
        <div>2014-11-24 18:05:33,820 INFO
           [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
          (org.ovirt.thread.pool-8-thread-45) [287d570d] FINISH,
          SetVdsStatusVDSCommand, log id: 1cbc7311</div>
        <div>2014-11-24 18:05:34,086 INFO
           [org.ovirt.engine.core.bll.ActivateVdsCommand]
          (org.ovirt.thread.pool-8-thread-45) Activate finished. Lock
          released. Monitoring can run now for host Compute5 from
          data-center SV_WTC</div>
        <div>2014-11-24 18:05:34,088 INFO
           [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
          (org.ovirt.thread.pool-8-thread-45) Correlation ID: 287d570d,
          Job ID: 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call Stack:
          null, Custom Event ID: -1, Message: Host Compute5 was
          activated by admin.</div>
        <div>2014-11-24 18:05:34,090 INFO
           [org.ovirt.engine.core.bll.ActivateVdsCommand]
          (org.ovirt.thread.pool-8-thread-45) Lock freed to object
          EngineLock [exclusiveLocks= key:
          0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS</div>
        <div>, sharedLocks= ]</div>
        <div>2014-11-24 18:05:35,792 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
          (DefaultQuartzScheduler_Worker-55) [3706e836] START,
          GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
          33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832</div>
        <div>2014-11-24 18:05:37,064 INFO
           [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
          (DefaultQuartzScheduler_Worker-69) START,
          GetHardwareInfoVDSCommand(HostName = Compute5, HostId =
          0bf6b00f-7947-4411-b55a-cc5eea2b381a,
          vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log
          id: 6d560cc2</div>
        <div>2014-11-24 18:05:37,074 INFO
           [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
          (DefaultQuartzScheduler_Worker-69) FINISH,
          GetHardwareInfoVDSCommand, log id: 6d560cc2</div>
        <div>2014-11-24 18:05:37,093 WARN
           [org.ovirt.engine.core.vdsbroker.VdsManager]
          (DefaultQuartzScheduler_Worker-69) Host Compute5 is running
          with disabled SELinux.</div>
        <div>2014-11-24 18:05:37,127 INFO
           [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
          (DefaultQuartzScheduler_Worker-69) [2b4a51cf] Running command:
          HandleVdsCpuFlagsOrClusterChangedCommand internal: true.
          Entities affected :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a
          Type: VDS</div>
        <div>2014-11-24 18:05:37,147 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
          (DefaultQuartzScheduler_Worker-69) [2b4a51cf] START,
          GlusterServersListVDSCommand(HostName = Compute5, HostId =
          0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87</div>
        <div>2014-11-24 18:05:37,164 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
          (DefaultQuartzScheduler_Worker-69) [2b4a51cf] FINISH,
          GlusterServersListVDSCommand, log id: 4faed87</div>
        <div>2014-11-24 18:05:37,189 INFO
           [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
          (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Running command:
          SetNonOperationalVdsCommand internal: true. Entities affected
          :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS</div>
        <div>2014-11-24 18:05:37,206 INFO
           [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
          (DefaultQuartzScheduler_Worker-69) [4a84c4e5] START,
          SetVdsStatusVDSCommand(HostName = Compute5, HostId =
          0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=NonOperational,
          nonOperationalReason=GLUSTER_COMMAND_FAILED,
          stopSpmFailureLogged=false), log id: fed5617</div>
        <div>2014-11-24 18:05:37,209 INFO
           [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
          (DefaultQuartzScheduler_Worker-69) [4a84c4e5] FINISH,
          SetVdsStatusVDSCommand, log id: fed5617</div>
        <div>2014-11-24 18:05:37,223 ERROR
          [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
          (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation ID:
          4a84c4e5, Job ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b, Call
          Stack: null, Custom Event ID: -1, Message: Gluster command
          [<UNKNOWN>] failed on server Compute5.</div>
        <div>2014-11-24 18:05:37,243 INFO
           [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
          (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation ID:
          null, Call Stack: null, Custom Event ID: -1, Message: Status
          of host Compute5 was set to NonOperational.</div>
        <div>2014-11-24 18:05:37,272 INFO
           [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
          (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Running command:
          HandleVdsVersionCommand internal: true. Entities affected :
           ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS</div>
        <div>2014-11-24 18:05:37,274 INFO
           [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
          (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host
          0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5 is already in
          NonOperational status for reason GLUSTER_COMMAND_FAILED.
          SetNonOperationalVds command is skipped.</div>
        <div>2014-11-24 18:05:38,065 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
          (DefaultQuartzScheduler_Worker-55) [3706e836] FINISH,
          GlusterVolumesListVDSCommand, return:
          {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4e72a1b1},
          log id: 48a0c832</div>
        <div>2014-11-24 18:05:43,243 INFO
           [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
          (DefaultQuartzScheduler_Worker-35) START,
          GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
          33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc</div>
        <div>^C</div>
        <div>[root@ccr01 ~]#</div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div>Punit</div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------080701090102060806020404--