[ovirt-users] Gluster command [<UNKNOWN>] failed on server...

Kanagaraj kmayilsa at redhat.com
Tue Nov 25 04:12:56 UTC 2014


Do you see any errors in 
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log or vdsm.log when the 
service is trying to start automatically after the reboot?

Thanks,
Kanagaraj

On 11/24/2014 08:13 PM, Punit Dambiwal wrote:
> Hi Kanagaraj,
>
> Yes...once i will start the gluster service and then vdsmd ...the host 
> can connect to cluster...but the question is why it's not started even 
> it has chkconfig enabled...
>
> I have tested it in two host cluster environment...(Centos 6.6 and 
> centos 7.0) on both hypervisior cluster..it's failed to reconnect in 
> to cluster after reboot....
>
> In both the environment glusterd enabled for next boot....but it's 
> failed with the same error....seems it's bug in either gluster or Ovirt ??
>
> Please help me to find the workaround here if can not resolve it...as 
> without this the Host machine can not connect after reboot....that 
> means engine will consider it as down and every time need to manually 
> start the gluster service and vdsmd... ??
>
> Thanks,
> Punit
>
> On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj <kmayilsa at redhat.com 
> <mailto:kmayilsa at redhat.com>> wrote:
>
>     From vdsm.log "error: Connection failed. Please check if gluster
>     daemon is operational."
>
>     Starting glusterd service should fix this issue. 'service glusterd
>     start'
>     But i am wondering why the glusterd was not started automatically
>     after the reboot.
>
>     Thanks,
>     Kanagaraj
>
>
>
>     On 11/24/2014 07:18 PM, Punit Dambiwal wrote:
>>     Hi Kanagaraj,
>>
>>     Please find the attached VDSM logs :-
>>
>>     ----------------
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>>     Owner.cancelAll requests {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
>>     Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref 0 aborting False
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
>>     Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
>>     init -> state preparing
>>     Thread-13::INFO::2014-11-24
>>     21:41:32,393::logUtils::44::dispatcher::(wrapper) Run and
>>     protect: repoStats(options=None)
>>     Thread-13::INFO::2014-11-24
>>     21:41:32,393::logUtils::47::dispatcher::(wrapper) Run and
>>     protect: repoStats, Return response: {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
>>     Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
>>     Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
>>     preparing -> state finished
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
>>     Owner.releaseAll requests {} resources {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>>     Owner.cancelAll requests {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
>>     Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref 0 aborting False
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,550::BindingXMLRPC::1132::vds::(wrapper) client
>>     [10.10.10.2]::call getCapabilities with () {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,553::utils::738::root::(execCmd) /sbin/ip route show to
>>     0.0.0.0/0 <http://0.0.0.0/0> table all (cwd None)
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,560::utils::758::root::(execCmd) SUCCESS: <err> = '';
>>     <rc> = 0
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,588::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,592::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift-object',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,593::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift-plugin',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift-account',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift-proxy',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift-doc',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
>>     ('gluster-swift-container',) not found
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
>>     ('glusterfs-geo-replication',) not found
>>     Thread-13::DEBUG::2014-11-24 21:41:41,600::caps::646::root::(get)
>>     VirtioRNG DISABLED: libvirt version 0.10.2-29.el6_5.9 required >=
>>     0.10.2-31
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,603::BindingXMLRPC::1139::vds::(wrapper) return
>>     getCapabilities with {'status': {'message': 'Done', 'code': 0},
>>     'info': {'HBAInventory': {'iSCSI': [{'InitiatorName':
>>     'iqn.1994-05.com.redhat:32151ce183c8'}], 'FC': []}, 'packages2':
>>     {'kernel': {'release': '431.el6.x86_64', 'buildtime':
>>     1385061309.0, 'version': '2.6.32'}, 'glusterfs-rdma': {'release':
>>     '1.el6', 'buildtime': 1403622628L, 'version': '3.5.1'},
>>     'glusterfs-fuse': {'release': '1.el6', 'buildtime': 1403622628L,
>>     'version': '3.5.1'}, 'spice-server': {'release': '6.el6_5.2',
>>     'buildtime': 1402324637L, 'version': '0.12.4'}, 'vdsm':
>>     {'release': '1.gitdb83943.el6', 'buildtime': 1412784567L,
>>     'version': '4.16.7'}, 'qemu-kvm': {'release': '2.415.el6_5.10',
>>     'buildtime': 1402435700L, 'version': '0.12.1.2'}, 'qemu-img':
>>     {'release': '2.415.el6_5.10', 'buildtime': 1402435700L,
>>     'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.9',
>>     'buildtime': 1402404612L, 'version': '0.10.2'}, 'glusterfs':
>>     {'release': '1.el6', 'buildtime': 1403622628L, 'version':
>>     '3.5.1'}, 'mom': {'release': '2.el6', 'buildtime': 1403794344L,
>>     'version': '0.4.1'}, 'glusterfs-server': {'release': '1.el6',
>>     'buildtime': 1403622628L, 'version': '3.5.1'}},
>>     'numaNodeDistance': {'1': [20, 10], '0': [10, 20]}, 'cpuModel':
>>     'Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz', 'liveMerge':
>>     'false', 'hooks': {}, 'cpuSockets': '2', 'vmTypes': ['kvm'],
>>     'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols':
>>     ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': u'bond0.10',
>>     'addr': '43.252.176.16', 'bridged': False, 'ipv6addrs':
>>     ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500', 'bootproto4':
>>     'none', 'netmask': '255.255.255.0', 'ipv4addrs':
>>     ['43.252.176.16/24' <http://43.252.176.16/24%27>], 'interface':
>>     u'bond0.10', 'ipv6gateway': '::', 'gateway': '43.25.17.1'},
>>     'Internal': {'iface': 'Internal', 'addr': '', 'cfg': {'DEFROUTE':
>>     'no', 'HOTPLUG': 'no', 'MTU': '9000', 'DELAY': '0',
>>     'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off',
>>     'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
>>     'bridged': True, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
>>     'gateway': '', 'bootproto4': 'none', 'netmask': '', 'stp': 'off',
>>     'ipv4addrs': [], 'mtu': '9000', 'ipv6gateway': '::', 'ports':
>>     ['bond1.100']}, 'storage': {'iface': u'bond1', 'addr':
>>     '10.10.10.6', 'bridged': False, 'ipv6addrs':
>>     ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000', 'bootproto4':
>>     'none', 'netmask': '255.255.255.0', 'ipv4addrs': ['10.10.10.6/24'
>>     <http://10.10.10.6/24%27>], 'interface': u'bond1', 'ipv6gateway':
>>     '::', 'gateway': ''}, 'VMNetwork': {'iface': 'VMNetwork', 'addr':
>>     '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU': '1500',
>>     'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP':
>>     'off', 'DEVICE': 'VMNetwork', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
>>     'bridged': True, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'],
>>     'gateway': '', 'bootproto4': 'none', 'netmask': '', 'stp': 'off',
>>     'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports':
>>     ['bond0.36']}}, 'bridges': {'Internal': {'addr': '', 'cfg':
>>     {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU': '9000', 'DELAY': '0',
>>     'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off',
>>     'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
>>     'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000',
>>     'netmask': '', 'stp': 'off', 'ipv4addrs': [], 'ipv6gateway':
>>     '::', 'gateway': '', 'opts': {'topology_change_detected': '0',
>>     'multicast_last_member_count': '2', 'hash_elasticity': '4',
>>     'multicast_query_response_interval': '999', 'multicast_snooping':
>>     '1', 'multicast_startup_query_interval': '3124', 'hello_timer':
>>     '31', 'multicast_querier_interval': '25496', 'max_age': '1999',
>>     'hash_max': '512', 'stp_state': '0', 'root_id':
>>     '8000.001018cddaac', 'priority': '32768',
>>     'multicast_membership_interval': '25996', 'root_path_cost': '0',
>>     'root_port': '0', 'multicast_querier': '0',
>>     'multicast_startup_query_count': '2', 'hello_time': '199',
>>     'topology_change': '0', 'bridge_id': '8000.001018cddaac',
>>     'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer':
>>     '31', 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0',
>>     'multicast_query_interval': '12498',
>>     'multicast_last_member_interval': '99', 'multicast_router': '1',
>>     'forward_delay': '0'}, 'ports': ['bond1.100']}, 'VMNetwork':
>>     {'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU':
>>     '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
>>     'STP': 'off', 'DEVICE': 'VMNetwork', 'TYPE': 'Bridge', 'ONBOOT':
>>     'no'}, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
>>     '1500', 'netmask': '', 'stp': 'off', 'ipv4addrs': [],
>>     'ipv6gateway': '::', 'gateway': '', 'opts':
>>     {'topology_change_detected': '0', 'multicast_last_member_count':
>>     '2', 'hash_elasticity': '4', 'multicast_query_response_interval':
>>     '999', 'multicast_snooping': '1',
>>     'multicast_startup_query_interval': '3124', 'hello_timer': '131',
>>     'multicast_querier_interval': '25496', 'max_age': '1999',
>>     'hash_max': '512', 'stp_state': '0', 'root_id':
>>     '8000.60eb6920b46c', 'priority': '32768',
>>     'multicast_membership_interval': '25996', 'root_path_cost': '0',
>>     'root_port': '0', 'multicast_querier': '0',
>>     'multicast_startup_query_count': '2', 'hello_time': '199',
>>     'topology_change': '0', 'bridge_id': '8000.60eb6920b46c',
>>     'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer':
>>     '31', 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0',
>>     'multicast_query_interval': '12498',
>>     'multicast_last_member_interval': '99', 'multicast_router': '1',
>>     'forward_delay': '0'}, 'ports': ['bond0.36']}}, 'uuid':
>>     '44454C4C-4C00-1057-8053-B7C04F504E31', 'lastClientIface':
>>     'bond1', 'nics': {'eth3': {'permhwaddr': '00:10:18:cd:da:ae',
>>     'addr': '', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU':
>>     '9000', 'HWADDR': '00:10:18:cd:da:ae', 'MASTER': 'bond1',
>>     'DEVICE': 'eth3', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu':
>>     '9000', 'netmask': '', 'ipv4addrs': [], 'hwaddr':
>>     '00:10:18:cd:da:ac', 'speed': 1000}, 'eth2': {'permhwaddr':
>>     '00:10:18:cd:da:ac', 'addr': '', 'cfg': {'SLAVE': 'yes',
>>     'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR':
>>     '00:10:18:cd:da:ac', 'MASTER': 'bond1', 'DEVICE': 'eth2',
>>     'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu': '9000', 'netmask': '',
>>     'ipv4addrs': [], 'hwaddr': '00:10:18:cd:da:ac', 'speed': 1000},
>>     'eth1': {'permhwaddr': '60:eb:69:20:b4:6d', 'addr': '', 'cfg':
>>     {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR':
>>     '60:eb:69:20:b4:6d', 'MASTER': 'bond0', 'DEVICE': 'eth1',
>>     'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '',
>>     'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'speed': 1000},
>>     'eth0': {'permhwaddr': '60:eb:69:20:b4:6c', 'addr': '', 'cfg':
>>     {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR':
>>     '60:eb:69:20:b4:6c', 'MASTER': 'bond0', 'DEVICE': 'eth0',
>>     'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '',
>>     'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'speed': 1000}},
>>     'software_revision': '1', 'clusterLevels': ['3.0', '3.1', '3.2',
>>     '3.3', '3.4', '3.5'], 'cpuFlags':
>>     u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
>>     'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:32151ce183c8',
>>     'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1',
>>     '3.2', '3.3', '3.4', '3.5'], 'autoNumaBalancing': 2,
>>     'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg':
>>     {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr':
>>     '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {'HOTPLUG':
>>     'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BONDING_OPTS':
>>     'mode=4 miimon=100', 'DEVICE': 'bond0', 'ONBOOT': 'yes'},
>>     'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500',
>>     'netmask': '', 'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c',
>>     'slaves': ['eth0', 'eth1'], 'opts': {'miimon': '100', 'mode':
>>     '4'}}, 'bond1': {'addr': '10.10.10.6', 'cfg': {'DEFROUTE': 'no',
>>     'IPADDR': '10.10.10.6', 'HOTPLUG': 'no', 'MTU': '9000',
>>     'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO':
>>     'none', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond1',
>>     'ONBOOT': 'no'}, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
>>     'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4addrs':
>>     ['10.10.10.6/24' <http://10.10.10.6/24%27>], 'hwaddr':
>>     '00:10:18:cd:da:ac', 'slaves': ['eth2', 'eth3'], 'opts':
>>     {'miimon': '100', 'mode': '4'}}, 'bond2': {'addr': '', 'cfg': {},
>>     'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr':
>>     '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu':
>>     '1500', 'netmask': '', 'slaves': [], 'hwaddr':
>>     '00:00:00:00:00:00'}}, 'software_version': '4.16', 'memSize':
>>     '24019', 'cpuSpeed': '2667.000', 'numaNodes': {u'1':
>>     {'totalMemory': '12288', 'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20,
>>     21, 22, 23]}, u'0': {'totalMemory': '12278', 'cpus': [0, 1, 2, 3,
>>     4, 5, 12, 13, 14, 15, 16, 17]}}, 'version_name': 'Snow Man',
>>     'vlans': {'bond0.10': {'iface': 'bond0', 'addr': '43.25.17.16',
>>     'cfg': {'DEFROUTE': 'yes', 'VLAN': 'yes', 'IPADDR':
>>     '43.25.17.16', 'HOTPLUG': 'no', 'GATEWAY': '43.25.17.1',
>>     'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO':
>>     'none', 'DEVICE': 'bond0.10', 'MTU': '1500', 'ONBOOT': 'yes'},
>>     'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 10,
>>     'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4addrs':
>>     ['43.25.17.16/24'] <http://43.25.17.16/24%27%5D>}, 'bond0.36':
>>     {'iface': 'bond0', 'addr': '', 'cfg': {'BRIDGE': 'VMNetwork',
>>     'VLAN': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED':
>>     'no', 'DEVICE': 'bond0.36', 'ONBOOT': 'no'}, 'ipv6addrs':
>>     ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 36, 'mtu': '1500',
>>     'netmask': '', 'ipv4addrs': []}, 'bond1.100': {'iface': 'bond1',
>>     'addr': '', 'cfg': {'BRIDGE': 'Internal', 'VLAN': 'yes',
>>     'HOTPLUG': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'DEVICE':
>>     'bond1.100', 'ONBOOT': 'no'}, 'ipv6addrs':
>>     ['fe80::210:18ff:fecd:daac/64'], 'vlanid': 100, 'mtu': '9000',
>>     'netmask': '', 'ipv4addrs': []}}, 'cpuCores': '12', 'kvmEnabled':
>>     'true', 'guestOverhead': '65', 'cpuThreads': '24',
>>     'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0',
>>     u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0',
>>     u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem':
>>     {'release': '5.el6.centos.11.1', 'version': '6', 'name': 'RHEL'},
>>     'lastClient': '10.10.10.2'}}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,620::BindingXMLRPC::1132::vds::(wrapper) client
>>     [10.10.10.2]::call getHardwareInfo with () {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,621::BindingXMLRPC::1139::vds::(wrapper) return
>>     getHardwareInfo with {'status': {'message': 'Done', 'code': 0},
>>     'info': {'systemProductName': 'CS24-TY', 'systemSerialNumber':
>>     '7LWSPN1', 'systemFamily': 'Server', 'systemVersion': 'A00',
>>     'systemUUID': '44454c4c-4c00-1057-8053-b7c04f504e31',
>>     'systemManufacturer': 'Dell'}}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:41,733::BindingXMLRPC::1132::vds::(wrapper) client
>>     [10.10.10.2]::call hostsList with () {} flowID [222e8036]
>>     Thread-13::ERROR::2014-11-24
>>     21:41:44,753::BindingXMLRPC::1148::vds::(wrapper) vdsm exception
>>     occured
>>     Traceback (most recent call last):
>>       File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, in wrapper
>>         res = f(*args, **kwargs)
>>       File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>>         rv = func(*args, **kwargs)
>>       File "/usr/share/vdsm/gluster/api.py", line 251, in hostsList
>>         return {'hosts': self.svdsmProxy.glusterPeerStatus()}
>>       File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
>>         return callMethod()
>>       File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>>         **kwargs)
>>       File "<string>", line 2, in glusterPeerStatus
>>       File "/usr/lib64/python2.6/multiprocessing/managers.py", line
>>     740, in _callmethod
>>         raise convert_to_error(kind, result)
>>     GlusterCmdExecFailedException: Command execution failed
>>     error: Connection failed. Please check if gluster daemon is
>>     operational.
>>     return code: 1
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
>>     Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
>>     init -> state preparing
>>     Thread-13::INFO::2014-11-24
>>     21:41:50,950::logUtils::44::dispatcher::(wrapper) Run and
>>     protect: repoStats(options=None)
>>     Thread-13::INFO::2014-11-24
>>     21:41:50,950::logUtils::47::dispatcher::(wrapper) Run and
>>     protect: repoStats, Return response: {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
>>     Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
>>     Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
>>     preparing -> state finished
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
>>     Owner.releaseAll requests {} resources {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>>     Owner.cancelAll requests {}
>>     Thread-13::DEBUG::2014-11-24
>>     21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
>>     Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref 0 aborting False
>>     -------------------------------
>>
>>     [root at compute4 ~]# service glusterd status
>>     glusterd is stopped
>>     [root at compute4 ~]# chkconfig --list | grep glusterd
>>     glusterd        0:off   1:off   2:on  3:on    4:on    5:on    6:off
>>     [root at compute4 ~]#
>>
>>     Thanks,
>>     Punit
>>
>>     On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj <kmayilsa at redhat.com
>>     <mailto:kmayilsa at redhat.com>> wrote:
>>
>>         Can you send the corresponding error in vdsm.log from the host?
>>
>>         Also check if glusterd service is running.
>>
>>         Thanks,
>>         Kanagaraj
>>
>>
>>         On 11/24/2014 03:39 PM, Punit Dambiwal wrote:
>>>         Hi,
>>>
>>>         After reboot my Hypervisior host can not activate again in
>>>         the cluster and failed with the following error :-
>>>
>>>         Gluster command [<UNKNOWN>] failed on server...
>>>
>>>         Engine logs :-
>>>
>>>         2014-11-24 18:05:28,397 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-64) START,
>>>         GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
>>>         33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90
>>>         2014-11-24 18:05:30,609 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-64) FINISH,
>>>         GlusterVolumesListVDSCommand, return:
>>>         {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d95203e0},
>>>         log id: 5f251c90
>>>         2014-11-24 18:05:33,768 INFO
>>>          [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>>         (ajp--127.0.0.1-8702-8) [287d570d] Lock Acquired to object
>>>         EngineLock [exclusiveLocks= key:
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
>>>         , sharedLocks= ]
>>>         2014-11-24 18:05:33,795 INFO
>>>          [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) [287d570d] Running
>>>         command: ActivateVdsCommand internal: false. Entities
>>>         affected :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
>>>         VDSAction group MANIPULATE_HOST with role type ADMIN
>>>         2014-11-24 18:05:33,796 INFO
>>>          [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) [287d570d] Before
>>>         acquiring lock in order to prevent monitoring for host
>>>         Compute5 from data-center SV_WTC
>>>         2014-11-24 18:05:33,797 INFO
>>>          [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) [287d570d] Lock
>>>         acquired, from now a monitoring of host will be skipped for
>>>         host Compute5 from data-center SV_WTC
>>>         2014-11-24 18:05:33,817 INFO
>>>          [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) [287d570d] START,
>>>         SetVdsStatusVDSCommand(HostName = Compute5, HostId =
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned,
>>>         nonOperationalReason=NONE, stopSpmFailureLogged=false), log
>>>         id: 1cbc7311
>>>         2014-11-24 18:05:33,820 INFO
>>>          [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) [287d570d] FINISH,
>>>         SetVdsStatusVDSCommand, log id: 1cbc7311
>>>         2014-11-24 18:05:34,086 INFO
>>>          [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) Activate finished. Lock
>>>         released. Monitoring can run now for host Compute5 from
>>>         data-center SV_WTC
>>>         2014-11-24 18:05:34,088 INFO
>>>          [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>         (org.ovirt.thread.pool-8-thread-45) Correlation ID:
>>>         287d570d, Job ID: 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call
>>>         Stack: null, Custom Event ID: -1, Message: Host Compute5 was
>>>         activated by admin.
>>>         2014-11-24 18:05:34,090 INFO
>>>          [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>>         (org.ovirt.thread.pool-8-thread-45) Lock freed to object
>>>         EngineLock [exclusiveLocks= key:
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
>>>         , sharedLocks= ]
>>>         2014-11-24 18:05:35,792 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-55) [3706e836] START,
>>>         GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
>>>         33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832
>>>         2014-11-24 18:05:37,064 INFO
>>>          [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-69) START,
>>>         GetHardwareInfoVDSCommand(HostName = Compute5, HostId =
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a,
>>>         vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]),
>>>         log id: 6d560cc2
>>>         2014-11-24 18:05:37,074 INFO
>>>          [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-69) FINISH,
>>>         GetHardwareInfoVDSCommand, log id: 6d560cc2
>>>         2014-11-24 18:05:37,093 WARN
>>>          [org.ovirt.engine.core.vdsbroker.VdsManager]
>>>         (DefaultQuartzScheduler_Worker-69) Host Compute5 is running
>>>         with disabled SELinux.
>>>         2014-11-24 18:05:37,127 INFO
>>>          [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [2b4a51cf] Running
>>>         command: HandleVdsCpuFlagsOrClusterChangedCommand internal:
>>>         true. Entities affected :  ID:
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
>>>         2014-11-24 18:05:37,147 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [2b4a51cf] START,
>>>         GlusterServersListVDSCommand(HostName = Compute5, HostId =
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87
>>>         2014-11-24 18:05:37,164 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [2b4a51cf] FINISH,
>>>         GlusterServersListVDSCommand, log id: 4faed87
>>>         2014-11-24 18:05:37,189 INFO
>>>          [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Running
>>>         command: SetNonOperationalVdsCommand internal: true.
>>>         Entities affected :  ID:
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
>>>         2014-11-24 18:05:37,206 INFO
>>>          [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [4a84c4e5] START,
>>>         SetVdsStatusVDSCommand(HostName = Compute5, HostId =
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=NonOperational,
>>>         nonOperationalReason=GLUSTER_COMMAND_FAILED,
>>>         stopSpmFailureLogged=false), log id: fed5617
>>>         2014-11-24 18:05:37,209 INFO
>>>          [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [4a84c4e5] FINISH,
>>>         SetVdsStatusVDSCommand, log id: fed5617
>>>         2014-11-24 18:05:37,223 ERROR
>>>         [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>         (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation
>>>         ID: 4a84c4e5, Job ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b,
>>>         Call Stack: null, Custom Event ID: -1, Message: Gluster
>>>         command [<UNKNOWN>] failed on server Compute5.
>>>         2014-11-24 18:05:37,243 INFO
>>>          [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>         (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation
>>>         ID: null, Call Stack: null, Custom Event ID: -1, Message:
>>>         Status of host Compute5 was set to NonOperational.
>>>         2014-11-24 18:05:37,272 INFO
>>>          [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
>>>         (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Running
>>>         command: HandleVdsVersionCommand internal: true. Entities
>>>         affected :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
>>>         2014-11-24 18:05:37,274 INFO
>>>          [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>         (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host
>>>         0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5 is already
>>>         in NonOperational status for reason GLUSTER_COMMAND_FAILED.
>>>         SetNonOperationalVds command is skipped.
>>>         2014-11-24 18:05:38,065 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-55) [3706e836] FINISH,
>>>         GlusterVolumesListVDSCommand, return:
>>>         {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4e72a1b1},
>>>         log id: 48a0c832
>>>         2014-11-24 18:05:43,243 INFO
>>>          [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>>         (DefaultQuartzScheduler_Worker-35) START,
>>>         GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
>>>         33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc
>>>         ^C
>>>         [root at ccr01 ~]#
>>>
>>>         Thanks,
>>>         Punit
>>>
>>>
>>>         _______________________________________________
>>>         Users mailing list
>>>         Users at ovirt.org  <mailto:Users at ovirt.org>
>>>         http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20141125/03748e00/attachment-0001.html>


More information about the Users mailing list