<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Do you see any errors in
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log or vdsm.log when
the service is trying to start automatically after the reboot?<br>
<br>
Thanks,<br>
Kanagaraj<br>
<br>
<div class="moz-cite-prefix">On 11/24/2014 08:13 PM, Punit Dambiwal
wrote:<br>
</div>
<blockquote
cite="mid:CAGZcrBmhGjZpN=fDZ+svTXD9V2sk=M45Jn3nJxwhKj38EODwrA@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Kanagaraj,
<div><br>
</div>
<div>Yes...once i will start the gluster service and then vdsmd
...the host can connect to cluster...but the question is why
it's not started even it has chkconfig enabled...</div>
<div><br>
</div>
<div>I have tested it in two host cluster environment...(Centos
6.6 and centos 7.0) on both hypervisior cluster..it's failed
to reconnect in to cluster after reboot....</div>
<div><br>
</div>
<div>In both the environment glusterd enabled for next
boot....but it's failed with the same error....seems it's bug
in either gluster or Ovirt ??</div>
<div><br>
</div>
<div>Please help me to find the workaround here if can not
resolve it...as without this the Host machine can not connect
after reboot....that means engine will consider it as down and
every time need to manually start the gluster service and
vdsmd... ??</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Punit</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Nov 24, 2014 at 10:20 PM,
Kanagaraj <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:kmayilsa@redhat.com" target="_blank">kmayilsa@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> From vdsm.log "error:
Connection failed. Please check if gluster daemon is
operational."<br>
<br>
Starting glusterd service should fix this issue. 'service
glusterd start'<br>
But i am wondering why the glusterd was not started
automatically after the reboot.<br>
<br>
Thanks,<br>
Kanagaraj
<div>
<div class="h5"><br>
<br>
<br>
<div>On 11/24/2014 07:18 PM, Punit Dambiwal wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Kanagaraj,
<div><br>
</div>
<div>Please find the attached VDSM logs :- </div>
<div><br>
</div>
<div>----------------</div>
<div>
<div>Thread-13::DEBUG::2014-11-24
21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref
0 aborting False</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving
from state init -> state preparing</div>
<div>Thread-13::<a moz-do-not-send="true">INFO::2014-11-24</a>
21:41:32,393::logUtils::44::dispatcher::(wrapper)
Run and protect: repoStats(options=None)</div>
<div>Thread-13::<a moz-do-not-send="true">INFO::2014-11-24</a>
21:41:32,393::logUtils::47::dispatcher::(wrapper)
Run and protect: repoStats, Return response:
{}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished:
{}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving
from state preparing -> state finished</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref
0 aborting False</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,550::BindingXMLRPC::1132::vds::(wrapper)
client [10.10.10.2]::call getCapabilities with
() {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,553::utils::738::root::(execCmd)
/sbin/ip route show to <a
moz-do-not-send="true"
href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a>
table all (cwd None)</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,560::utils::758::root::(execCmd)
SUCCESS: <err> = ''; <rc> = 0</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,588::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift',) not found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,592::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift-object',) not
found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,593::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift-plugin',) not
found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,598::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift-account',) not
found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,598::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift-proxy',) not found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,598::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift-doc',) not found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,599::caps::728::root::(_getKeyPackages)
rpm package ('gluster-swift-container',) not
found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,599::caps::728::root::(_getKeyPackages)
rpm package ('glusterfs-geo-replication',) not
found</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,600::caps::646::root::(get) VirtioRNG
DISABLED: libvirt version 0.10.2-29.el6_5.9
required >= 0.10.2-31</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,603::BindingXMLRPC::1139::vds::(wrapper)
return getCapabilities with {'status':
{'message': 'Done', 'code': 0}, 'info':
{'HBAInventory': {'iSCSI': [{'InitiatorName':
'iqn.1994-05.com.redhat:32151ce183c8'}], 'FC':
[]}, 'packages2': {'kernel': {'release':
'431.el6.x86_64', 'buildtime': 1385061309.0,
'version': '2.6.32'}, 'glusterfs-rdma':
{'release': '1.el6', 'buildtime': 1403622628L,
'version': '3.5.1'}, 'glusterfs-fuse':
{'release': '1.el6', 'buildtime': 1403622628L,
'version': '3.5.1'}, 'spice-server':
{'release': '6.el6_5.2', 'buildtime':
1402324637L, 'version': '0.12.4'}, 'vdsm':
{'release': '1.gitdb83943.el6', 'buildtime':
1412784567L, 'version': '4.16.7'}, 'qemu-kvm':
{'release': '2.415.el6_5.10', 'buildtime':
1402435700L, 'version': '0.12.1.2'},
'qemu-img': {'release': '2.415.el6_5.10',
'buildtime': 1402435700L, 'version':
'0.12.1.2'}, 'libvirt': {'release':
'29.el6_5.9', 'buildtime': 1402404612L,
'version': '0.10.2'}, 'glusterfs': {'release':
'1.el6', 'buildtime': 1403622628L, 'version':
'3.5.1'}, 'mom': {'release': '2.el6',
'buildtime': 1403794344L, 'version': '0.4.1'},
'glusterfs-server': {'release': '1.el6',
'buildtime': 1403622628L, 'version':
'3.5.1'}}, 'numaNodeDistance': {'1': [20, 10],
'0': [10, 20]}, 'cpuModel': 'Intel(R) Xeon(R)
CPU X5650 @ 2.67GHz', 'liveMerge':
'false', 'hooks': {}, 'cpuSockets': '2',
'vmTypes': ['kvm'], 'selinux': {'mode': '1'},
'kdumpStatus': 0, 'supportedProtocols':
['2.2', '2.3'], 'networks': {'ovirtmgmt':
{'iface': u'bond0.10', 'addr':
'43.252.176.16', 'bridged': False,
'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'],
'mtu': '1500', 'bootproto4': 'none',
'netmask': '255.255.255.0', 'ipv4addrs': ['<a
moz-do-not-send="true"
href="http://43.252.176.16/24%27"
target="_blank">43.252.176.16/24'</a>],
'interface': u'bond0.10', 'ipv6gateway': '::',
'gateway': '43.25.17.1'}, 'Internal':
{'iface': 'Internal', 'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU':
'9000', 'DELAY': '0', 'NM_CONTROLLED': 'no',
'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE':
'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
'bridged': True, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'gateway':
'', 'bootproto4': 'none', 'netmask': '',
'stp': 'off', 'ipv4addrs': [], 'mtu': '9000',
'ipv6gateway': '::', 'ports': ['bond1.100']},
'storage': {'iface': u'bond1', 'addr':
'10.10.10.6', 'bridged': False, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'mtu':
'9000', 'bootproto4': 'none', 'netmask':
'255.255.255.0', 'ipv4addrs': ['<a
moz-do-not-send="true"
href="http://10.10.10.6/24%27"
target="_blank">10.10.10.6/24'</a>],
'interface': u'bond1', 'ipv6gateway': '::',
'gateway': ''}, 'VMNetwork': {'iface':
'VMNetwork', 'addr': '', 'cfg': {'DEFROUTE':
'no', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY':
'0', 'NM_CONTROLLED': 'no', 'BOOTPROTO':
'none', 'STP': 'off', 'DEVICE': 'VMNetwork',
'TYPE': 'Bridge', 'ONBOOT': 'no'}, 'bridged':
True, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'gateway':
'', 'bootproto4': 'none', 'netmask': '',
'stp': 'off', 'ipv4addrs': [], 'mtu': '1500',
'ipv6gateway': '::', 'ports': ['bond0.36']}},
'bridges': {'Internal': {'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU':
'9000', 'DELAY': '0', 'NM_CONTROLLED': 'no',
'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE':
'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'},
'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
'mtu': '9000', 'netmask': '', 'stp': 'off',
'ipv4addrs': [], 'ipv6gateway': '::',
'gateway': '', 'opts':
{'topology_change_detected': '0',
'multicast_last_member_count': '2',
'hash_elasticity': '4',
'multicast_query_response_interval': '999',
'multicast_snooping': '1',
'multicast_startup_query_interval': '3124',
'hello_timer': '31',
'multicast_querier_interval': '25496',
'max_age': '1999', 'hash_max': '512',
'stp_state': '0', 'root_id':
'8000.001018cddaac', 'priority': '32768',
'multicast_membership_interval': '25996',
'root_path_cost': '0', 'root_port': '0',
'multicast_querier': '0',
'multicast_startup_query_count': '2',
'hello_time': '199', 'topology_change': '0',
'bridge_id': '8000.001018cddaac',
'topology_change_timer': '0', 'ageing_time':
'29995', 'gc_timer': '31', 'group_addr':
'1:80:c2:0:0:0', 'tcn_timer': '0',
'multicast_query_interval': '12498',
'multicast_last_member_interval': '99',
'multicast_router': '1', 'forward_delay':
'0'}, 'ports': ['bond1.100']}, 'VMNetwork':
{'addr': '', 'cfg': {'DEFROUTE': 'no',
'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP': 'off', 'DEVICE': 'VMNetwork', 'TYPE':
'Bridge', 'ONBOOT': 'no'}, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
'1500', 'netmask': '', 'stp': 'off',
'ipv4addrs': [], 'ipv6gateway': '::',
'gateway': '', 'opts':
{'topology_change_detected': '0',
'multicast_last_member_count': '2',
'hash_elasticity': '4',
'multicast_query_response_interval': '999',
'multicast_snooping': '1',
'multicast_startup_query_interval': '3124',
'hello_timer': '131',
'multicast_querier_interval': '25496',
'max_age': '1999', 'hash_max': '512',
'stp_state': '0', 'root_id':
'8000.60eb6920b46c', 'priority': '32768',
'multicast_membership_interval': '25996',
'root_path_cost': '0', 'root_port': '0',
'multicast_querier': '0',
'multicast_startup_query_count': '2',
'hello_time': '199', 'topology_change': '0',
'bridge_id': '8000.60eb6920b46c',
'topology_change_timer': '0', 'ageing_time':
'29995', 'gc_timer': '31', 'group_addr':
'1:80:c2:0:0:0', 'tcn_timer': '0',
'multicast_query_interval': '12498',
'multicast_last_member_interval': '99',
'multicast_router': '1', 'forward_delay':
'0'}, 'ports': ['bond0.36']}}, 'uuid':
'44454C4C-4C00-1057-8053-B7C04F504E31',
'lastClientIface': 'bond1', 'nics': {'eth3':
{'permhwaddr': '00:10:18:cd:da:ae', 'addr':
'', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED':
'no', 'MTU': '9000', 'HWADDR':
'00:10:18:cd:da:ae', 'MASTER': 'bond1',
'DEVICE': 'eth3', 'ONBOOT': 'no'},
'ipv6addrs': [], 'mtu': '9000', 'netmask': '',
'ipv4addrs': [], 'hwaddr':
'00:10:18:cd:da:ac', 'speed': 1000}, 'eth2':
{'permhwaddr': '00:10:18:cd:da:ac', 'addr':
'', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED':
'no', 'MTU': '9000', 'HWADDR':
'00:10:18:cd:da:ac', 'MASTER': 'bond1',
'DEVICE': 'eth2', 'ONBOOT': 'no'},
'ipv6addrs': [], 'mtu': '9000', 'netmask': '',
'ipv4addrs': [], 'hwaddr':
'00:10:18:cd:da:ac', 'speed': 1000}, 'eth1':
{'permhwaddr': '60:eb:69:20:b4:6d', 'addr':
'', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED':
'no', 'MTU': '1500', 'HWADDR':
'60:eb:69:20:b4:6d', 'MASTER': 'bond0',
'DEVICE': 'eth1', 'ONBOOT': 'yes'},
'ipv6addrs': [], 'mtu': '1500', 'netmask': '',
'ipv4addrs': [], 'hwaddr':
'60:eb:69:20:b4:6c', 'speed': 1000}, 'eth0':
{'permhwaddr': '60:eb:69:20:b4:6c', 'addr':
'', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED':
'no', 'MTU': '1500', 'HWADDR':
'60:eb:69:20:b4:6c', 'MASTER': 'bond0',
'DEVICE': 'eth0', 'ONBOOT': 'yes'},
'ipv6addrs': [], 'mtu': '1500', 'netmask': '',
'ipv4addrs': [], 'hwaddr':
'60:eb:69:20:b4:6c', 'speed': 1000}},
'software_revision': '1', 'clusterLevels':
['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'],
'cpuFlags':
u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
'ISCSIInitiatorName':
'iqn.1994-05.com.redhat:32151ce183c8',
'netConfigDirty': 'False', 'supportedENGINEs':
['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'],
'autoNumaBalancing': 2, 'reservedMem': '321',
'bondings': {'bond4': {'addr': '', 'cfg': {},
'mtu': '1500', 'netmask': '', 'slaves': [],
'hwaddr': '00:00:00:00:00:00'}, 'bond0':
{'addr': '', 'cfg': {'HOTPLUG': 'no', 'MTU':
'1500', 'NM_CONTROLLED': 'no', 'BONDING_OPTS':
'mode=4 miimon=100', 'DEVICE': 'bond0',
'ONBOOT': 'yes'}, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
'1500', 'netmask': '', 'ipv4addrs': [],
'hwaddr': '60:eb:69:20:b4:6c', 'slaves':
['eth0', 'eth1'], 'opts': {'miimon': '100',
'mode': '4'}}, 'bond1': {'addr': '10.10.10.6',
'cfg': {'DEFROUTE': 'no', 'IPADDR':
'10.10.10.6', 'HOTPLUG': 'no', 'MTU': '9000',
'NM_CONTROLLED': 'no', 'NETMASK':
'255.255.255.0', 'BOOTPROTO': 'none',
'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE':
'bond1', 'ONBOOT': 'no'}, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'mtu':
'9000', 'netmask': '255.255.255.0',
'ipv4addrs': ['<a moz-do-not-send="true"
href="http://10.10.10.6/24%27"
target="_blank">10.10.10.6/24'</a>],
'hwaddr': '00:10:18:cd:da:ac', 'slaves':
['eth2', 'eth3'], 'opts': {'miimon': '100',
'mode': '4'}}, 'bond2': {'addr': '', 'cfg':
{}, 'mtu': '1500', 'netmask': '', 'slaves':
[], 'hwaddr': '00:00:00:00:00:00'}, 'bond3':
{'addr': '', 'cfg': {}, 'mtu': '1500',
'netmask': '', 'slaves': [], 'hwaddr':
'00:00:00:00:00:00'}}, 'software_version':
'4.16', 'memSize': '24019', 'cpuSpeed':
'2667.000', 'numaNodes': {u'1':
{'totalMemory': '12288', 'cpus': [6, 7, 8, 9,
10, 11, 18, 19, 20, 21, 22, 23]}, u'0':
{'totalMemory': '12278', 'cpus': [0, 1, 2, 3,
4, 5, 12, 13, 14, 15, 16, 17]}},
'version_name': 'Snow Man', 'vlans':
{'bond0.10': {'iface': 'bond0', 'addr':
'43.25.17.16', 'cfg': {'DEFROUTE': 'yes',
'VLAN': 'yes', 'IPADDR': '43.25.17.16',
'HOTPLUG': 'no', 'GATEWAY': '43.25.17.1',
'NM_CONTROLLED': 'no', 'NETMASK':
'255.255.255.0', 'BOOTPROTO': 'none',
'DEVICE': 'bond0.10', 'MTU': '1500', 'ONBOOT':
'yes'}, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid':
10, 'mtu': '1500', 'netmask': '255.255.255.0',
'ipv4addrs': ['<a moz-do-not-send="true"
href="http://43.25.17.16/24%27%5D"
target="_blank">43.25.17.16/24']</a>},
'bond0.36': {'iface': 'bond0', 'addr': '',
'cfg': {'BRIDGE': 'VMNetwork', 'VLAN': 'yes',
'HOTPLUG': 'no', 'MTU': '1500',
'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.36',
'ONBOOT': 'no'}, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid':
36, 'mtu': '1500', 'netmask': '', 'ipv4addrs':
[]}, 'bond1.100': {'iface': 'bond1', 'addr':
'', 'cfg': {'BRIDGE': 'Internal', 'VLAN':
'yes', 'HOTPLUG': 'no', 'MTU': '9000',
'NM_CONTROLLED': 'no', 'DEVICE': 'bond1.100',
'ONBOOT': 'no'}, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'vlanid':
100, 'mtu': '9000', 'netmask': '',
'ipv4addrs': []}}, 'cpuCores': '12',
'kvmEnabled': 'true', 'guestOverhead': '65',
'cpuThreads': '24', 'emulatedMachines':
[u'rhel6.5.0', u'pc', u'rhel6.4.0',
u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0',
u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4',
u'rhel5.4.0'], 'operatingSystem': {'release':
'5.el6.centos.11.1', 'version': '6', 'name':
'RHEL'}, 'lastClient': '10.10.10.2'}}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,620::BindingXMLRPC::1132::vds::(wrapper)
client [10.10.10.2]::call getHardwareInfo with
() {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,621::BindingXMLRPC::1139::vds::(wrapper)
return getHardwareInfo with {'status':
{'message': 'Done', 'code': 0}, 'info':
{'systemProductName': 'CS24-TY',
'systemSerialNumber': '7LWSPN1',
'systemFamily': 'Server', 'systemVersion':
'A00', 'systemUUID':
'44454c4c-4c00-1057-8053-b7c04f504e31',
'systemManufacturer': 'Dell'}}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:41,733::BindingXMLRPC::1132::vds::(wrapper)
client [10.10.10.2]::call hostsList with () {}
flowID [222e8036]</div>
<div>Thread-13::ERROR::2014-11-24
21:41:44,753::BindingXMLRPC::1148::vds::(wrapper)
vdsm exception occured</div>
<div>Traceback (most recent call last):</div>
<div> File
"/usr/share/vdsm/rpc/BindingXMLRPC.py", line
1135, in wrapper</div>
<div> res = f(*args, **kwargs)</div>
<div> File "/usr/share/vdsm/gluster/api.py",
line 54, in wrapper</div>
<div> rv = func(*args, **kwargs)</div>
<div> File "/usr/share/vdsm/gluster/api.py",
line 251, in hostsList</div>
<div> return {'hosts':
self.svdsmProxy.glusterPeerStatus()}</div>
<div> File "/usr/share/vdsm/supervdsm.py", line
50, in __call__</div>
<div> return callMethod()</div>
<div> File "/usr/share/vdsm/supervdsm.py", line
48, in <lambda></div>
<div> **kwargs)</div>
<div> File "<string>", line 2, in
glusterPeerStatus</div>
<div> File
"/usr/lib64/python2.6/multiprocessing/managers.py",
line 740, in _callmethod</div>
<div> raise convert_to_error(kind, result)</div>
<div>GlusterCmdExecFailedException: Command
execution failed</div>
<div>error: Connection failed. Please check if
gluster daemon is operational.</div>
<div>return code: 1</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving
from state init -> state preparing</div>
<div>Thread-13::<a moz-do-not-send="true">INFO::2014-11-24</a>
21:41:50,950::logUtils::44::dispatcher::(wrapper)
Run and protect: repoStats(options=None)</div>
<div>Thread-13::<a moz-do-not-send="true">INFO::2014-11-24</a>
21:41:50,950::logUtils::47::dispatcher::(wrapper)
Run and protect: repoStats, Return response:
{}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished:
{}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving
from state preparing -> state finished</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}</div>
<div>Thread-13::DEBUG::2014-11-24
21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref
0 aborting False</div>
</div>
<div>-------------------------------</div>
<div><br>
</div>
<div>
<div>[root@compute4 ~]# service glusterd status</div>
<div>glusterd is stopped</div>
<div>[root@compute4 ~]# chkconfig --list | grep
glusterd</div>
<div>glusterd 0:off 1:off 2:on
3:on 4:on 5:on 6:off</div>
<div>[root@compute4 ~]#<br>
</div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Punit</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Nov 24, 2014 at
6:36 PM, Kanagaraj <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:kmayilsa@redhat.com"
target="_blank">kmayilsa@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Can you
send the corresponding error in vdsm.log
from the host?<br>
<br>
Also check if glusterd service is running.<br>
<br>
Thanks,<br>
Kanagaraj
<div>
<div><br>
<br>
<div>On 11/24/2014 03:39 PM, Punit
Dambiwal wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>After reboot my Hypervisior
host can not activate again in the
cluster and failed with the
following error :- </div>
<div><br>
</div>
<div>Gluster command
[<UNKNOWN>] failed on
server...<br>
</div>
<div><br>
</div>
<div>Engine logs :- </div>
<div><br>
</div>
<div>2014-11-24 18:05:28,397 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-64)
START,
GlusterVolumesListVDSCommand(HostName
= Compute4, HostId =
33648a90-200c-45ca-89d5-1ce305d79a6a),
log id: 5f251c90</div>
<div>2014-11-24 18:05:30,609 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-64)
FINISH,
GlusterVolumesListVDSCommand,
return:
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d95203e0},
log id: 5f251c90</div>
<div>2014-11-24 18:05:33,768 INFO
[org.ovirt.engine.core.bll.ActivateVdsCommand]
(ajp--127.0.0.1-8702-8) [287d570d]
Lock Acquired to object EngineLock
[exclusiveLocks= key:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
value: VDS</div>
<div>, sharedLocks= ]</div>
<div>2014-11-24 18:05:33,795 INFO
[org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-8-thread-45)
[287d570d] Running command:
ActivateVdsCommand internal:
false. Entities affected : ID:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDSAction group
MANIPULATE_HOST with role type
ADMIN</div>
<div>2014-11-24 18:05:33,796 INFO
[org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-8-thread-45)
[287d570d] Before acquiring lock
in order to prevent monitoring for
host Compute5 from data-center
SV_WTC</div>
<div>2014-11-24 18:05:33,797 INFO
[org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-8-thread-45)
[287d570d] Lock acquired, from now
a monitoring of host will be
skipped for host Compute5 from
data-center SV_WTC</div>
<div>2014-11-24 18:05:33,817 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-45)
[287d570d] START,
SetVdsStatusVDSCommand(HostName =
Compute5, HostId =
0bf6b00f-7947-4411-b55a-cc5eea2b381a,
status=Unassigned,
nonOperationalReason=NONE,
stopSpmFailureLogged=false), log
id: 1cbc7311</div>
<div>2014-11-24 18:05:33,820 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-45)
[287d570d] FINISH,
SetVdsStatusVDSCommand, log id:
1cbc7311</div>
<div>2014-11-24 18:05:34,086 INFO
[org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-8-thread-45)
Activate finished. Lock released.
Monitoring can run now for host
Compute5 from data-center SV_WTC</div>
<div>2014-11-24 18:05:34,088 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-45)
Correlation ID: 287d570d, Job ID:
5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a,
Call Stack: null, Custom Event ID:
-1, Message: Host Compute5 was
activated by admin.</div>
<div>2014-11-24 18:05:34,090 INFO
[org.ovirt.engine.core.bll.ActivateVdsCommand]
(org.ovirt.thread.pool-8-thread-45)
Lock freed to object EngineLock
[exclusiveLocks= key:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
value: VDS</div>
<div>, sharedLocks= ]</div>
<div>2014-11-24 18:05:35,792 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-55)
[3706e836] START,
GlusterVolumesListVDSCommand(HostName
= Compute4, HostId =
33648a90-200c-45ca-89d5-1ce305d79a6a),
log id: 48a0c832</div>
<div>2014-11-24 18:05:37,064 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler_Worker-69)
START,
GetHardwareInfoVDSCommand(HostName
= Compute5, HostId =
0bf6b00f-7947-4411-b55a-cc5eea2b381a,
vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]),
log id: 6d560cc2</div>
<div>2014-11-24 18:05:37,074 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler_Worker-69)
FINISH, GetHardwareInfoVDSCommand,
log id: 6d560cc2</div>
<div>2014-11-24 18:05:37,093 WARN
[org.ovirt.engine.core.vdsbroker.VdsManager]
(DefaultQuartzScheduler_Worker-69)
Host Compute5 is running with
disabled SELinux.</div>
<div>2014-11-24 18:05:37,127 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(DefaultQuartzScheduler_Worker-69)
[2b4a51cf] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand
internal: true. Entities affected
: ID:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDS</div>
<div>2014-11-24 18:05:37,147 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler_Worker-69)
[2b4a51cf] START,
GlusterServersListVDSCommand(HostName
= Compute5, HostId =
0bf6b00f-7947-4411-b55a-cc5eea2b381a),
log id: 4faed87</div>
<div>2014-11-24 18:05:37,164 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler_Worker-69)
[2b4a51cf] FINISH,
GlusterServersListVDSCommand, log
id: 4faed87</div>
<div>2014-11-24 18:05:37,189 INFO
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Running command:
SetNonOperationalVdsCommand
internal: true. Entities affected
: ID:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDS</div>
<div>2014-11-24 18:05:37,206 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69)
[4a84c4e5] START,
SetVdsStatusVDSCommand(HostName =
Compute5, HostId =
0bf6b00f-7947-4411-b55a-cc5eea2b381a,
status=NonOperational,
nonOperationalReason=GLUSTER_COMMAND_FAILED,
stopSpmFailureLogged=false), log
id: fed5617</div>
<div>2014-11-24 18:05:37,209 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69)
[4a84c4e5] FINISH,
SetVdsStatusVDSCommand, log id:
fed5617</div>
<div>2014-11-24 18:05:37,223 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Correlation ID:
4a84c4e5, Job ID:
4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b,
Call Stack: null, Custom Event ID:
-1, Message: Gluster command
[<UNKNOWN>] failed on server
Compute5.</div>
<div>2014-11-24 18:05:37,243 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Correlation ID: null,
Call Stack: null, Custom Event ID:
-1, Message: Status of host
Compute5 was set to
NonOperational.</div>
<div>2014-11-24 18:05:37,272 INFO
[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(DefaultQuartzScheduler_Worker-69)
[a0c8a7f] Running command:
HandleVdsVersionCommand internal:
true. Entities affected : ID:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDS</div>
<div>2014-11-24 18:05:37,274 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-69)
[a0c8a7f] Host
0bf6b00f-7947-4411-b55a-cc5eea2b381a
: Compute5 is already in
NonOperational status for reason
GLUSTER_COMMAND_FAILED.
SetNonOperationalVds command is
skipped.</div>
<div>2014-11-24 18:05:38,065 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-55)
[3706e836] FINISH,
GlusterVolumesListVDSCommand,
return:
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4e72a1b1},
log id: 48a0c832</div>
<div>2014-11-24 18:05:43,243 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-35)
START,
GlusterVolumesListVDSCommand(HostName
= Compute4, HostId =
33648a90-200c-45ca-89d5-1ce305d79a6a),
log id: 3ce13ebc</div>
<div>^C</div>
<div>[root@ccr01 ~]#</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Punit</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>