[ovirt-users] Gluster command [<UNKNOWN>] failed on server...
Kanagaraj Mayilsamy
kmayilsa at redhat.com
Tue Nov 25 17:59:59 UTC 2014
[+Gluster-users at gluster.org]
"Initialization of volume 'management' failed, review your volfile again", glusterd throws this error when the service is started automatically after the reboot. But the service is successfully started later manually by the user.
can somebody from gluster-users please help on this?
glusterfs version: 3.5.1
Thanks,
Kanagaraj
----- Original Message -----
> From: "Punit Dambiwal" <hypunit at gmail.com>
> To: "Kanagaraj" <kmayilsa at redhat.com>
> Cc: users at ovirt.org
> Sent: Tuesday, November 25, 2014 7:24:45 PM
> Subject: Re: [ovirt-users] Gluster command [<UNKNOWN>] failed on server...
>
> Hi Kanagraj,
>
> Please check the attached log files....i didn't find any thing special....
>
> On Tue, Nov 25, 2014 at 12:12 PM, Kanagaraj <kmayilsa at redhat.com> wrote:
>
> > Do you see any errors in
> > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log or vdsm.log when the
> > service is trying to start automatically after the reboot?
> >
> > Thanks,
> > Kanagaraj
> >
> >
> > On 11/24/2014 08:13 PM, Punit Dambiwal wrote:
> >
> > Hi Kanagaraj,
> >
> > Yes...once i will start the gluster service and then vdsmd ...the host
> > can connect to cluster...but the question is why it's not started even it
> > has chkconfig enabled...
> >
> > I have tested it in two host cluster environment...(Centos 6.6 and
> > centos 7.0) on both hypervisior cluster..it's failed to reconnect in to
> > cluster after reboot....
> >
> > In both the environment glusterd enabled for next boot....but it's
> > failed with the same error....seems it's bug in either gluster or Ovirt ??
> >
> > Please help me to find the workaround here if can not resolve it...as
> > without this the Host machine can not connect after reboot....that means
> > engine will consider it as down and every time need to manually start the
> > gluster service and vdsmd... ??
> >
> > Thanks,
> > Punit
> >
> > On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj <kmayilsa at redhat.com> wrote:
> >
> >> From vdsm.log "error: Connection failed. Please check if gluster daemon
> >> is operational."
> >>
> >> Starting glusterd service should fix this issue. 'service glusterd start'
> >> But i am wondering why the glusterd was not started automatically after
> >> the reboot.
> >>
> >> Thanks,
> >> Kanagaraj
> >>
> >>
> >>
> >> On 11/24/2014 07:18 PM, Punit Dambiwal wrote:
> >>
> >> Hi Kanagaraj,
> >>
> >> Please find the attached VDSM logs :-
> >>
> >> ----------------
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
> >> Owner.cancelAll requests {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
> >> Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref 0 aborting False
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
> >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state init ->
> >> state preparing
> >> Thread-13::INFO::2014-11-24
> >> 21:41:32,393::logUtils::44::dispatcher::(wrapper) Run and protect:
> >> repoStats(options=None)
> >> Thread-13::INFO::2014-11-24
> >> 21:41:32,393::logUtils::47::dispatcher::(wrapper) Run and protect:
> >> repoStats, Return response: {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
> >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
> >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state preparing
> >> ->
> >> state finished
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
> >> Owner.releaseAll requests {} resources {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
> >> Owner.cancelAll requests {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
> >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref 0 aborting False
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,550::BindingXMLRPC::1132::vds::(wrapper) client
> >> [10.10.10.2]::call
> >> getCapabilities with () {}
> >> Thread-13::DEBUG::2014-11-24 21:41:41,553::utils::738::root::(execCmd)
> >> /sbin/ip route show to 0.0.0.0/0 table all (cwd None)
> >> Thread-13::DEBUG::2014-11-24 21:41:41,560::utils::758::root::(execCmd)
> >> SUCCESS: <err> = ''; <rc> = 0
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,588::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,592::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift-object',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,593::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift-plugin',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift-account',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift-proxy',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift-doc',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
> >> ('gluster-swift-container',) not found
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
> >> ('glusterfs-geo-replication',) not found
> >> Thread-13::DEBUG::2014-11-24 21:41:41,600::caps::646::root::(get)
> >> VirtioRNG DISABLED: libvirt version 0.10.2-29.el6_5.9 required >=
> >> 0.10.2-31
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,603::BindingXMLRPC::1139::vds::(wrapper) return getCapabilities
> >> with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory':
> >> {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:32151ce183c8'}],
> >> 'FC':
> >> []}, 'packages2': {'kernel': {'release': '431.el6.x86_64', 'buildtime':
> >> 1385061309.0, 'version': '2.6.32'}, 'glusterfs-rdma': {'release': '1.el6',
> >> 'buildtime': 1403622628L, 'version': '3.5.1'}, 'glusterfs-fuse':
> >> {'release': '1.el6', 'buildtime': 1403622628L, 'version': '3.5.1'},
> >> 'spice-server': {'release': '6.el6_5.2', 'buildtime': 1402324637L,
> >> 'version': '0.12.4'}, 'vdsm': {'release': '1.gitdb83943.el6', 'buildtime':
> >> 1412784567L, 'version': '4.16.7'}, 'qemu-kvm': {'release':
> >> '2.415.el6_5.10', 'buildtime': 1402435700L, 'version': '0.12.1.2'},
> >> 'qemu-img': {'release': '2.415.el6_5.10', 'buildtime': 1402435700L,
> >> 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.9', 'buildtime':
> >> 1402404612L, 'version': '0.10.2'}, 'glusterfs': {'release': '1.el6',
> >> 'buildtime': 1403622628L, 'version': '3.5.1'}, 'mom': {'release': '2.el6',
> >> 'buildtime': 1403794344L, 'version': '0.4.1'}, 'glusterfs-server':
> >> {'release': '1.el6', 'buildtime': 1403622628L, 'version': '3.5.1'}},
> >> 'numaNodeDistance': {'1': [20, 10], '0': [10, 20]}, 'cpuModel': 'Intel(R)
> >> Xeon(R) CPU X5650 @ 2.67GHz', 'liveMerge': 'false', 'hooks':
> >> {},
> >> 'cpuSockets': '2', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'},
> >> 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks':
> >> {'ovirtmgmt': {'iface': u'bond0.10', 'addr': '43.252.176.16', 'bridged':
> >> False, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500',
> >> 'bootproto4': 'none', 'netmask': '255.255.255.0', 'ipv4addrs': ['
> >> 43.252.176.16/24' <http://43.252.176.16/24%27>], 'interface':
> >> u'bond0.10', 'ipv6gateway': '::', 'gateway': '43.25.17.1'}, 'Internal':
> >> {'iface': 'Internal', 'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG':
> >> 'no', 'MTU': '9000', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO':
> >> 'none', 'STP': 'off', 'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT':
> >> 'no'}, 'bridged': True, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
> >> 'gateway': '', 'bootproto4': 'none', 'netmask': '', 'stp': 'off',
> >> 'ipv4addrs': [], 'mtu': '9000', 'ipv6gateway': '::', 'ports':
> >> ['bond1.100']}, 'storage': {'iface': u'bond1', 'addr': '10.10.10.6',
> >> 'bridged': False, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], 'mtu':
> >> '9000', 'bootproto4': 'none', 'netmask': '255.255.255.0', 'ipv4addrs': ['
> >> 10.10.10.6/24' <http://10.10.10.6/24%27>], 'interface': u'bond1',
> >> 'ipv6gateway': '::', 'gateway': ''}, 'VMNetwork': {'iface': 'VMNetwork',
> >> 'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU': '1500',
> >> 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off',
> >> 'DEVICE': 'VMNetwork', 'TYPE': 'Bridge', 'ONBOOT': 'no'}, 'bridged': True,
> >> 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'gateway': '',
> >> 'bootproto4':
> >> 'none', 'netmask': '', 'stp': 'off', 'ipv4addrs': [], 'mtu': '1500',
> >> 'ipv6gateway': '::', 'ports': ['bond0.36']}}, 'bridges': {'Internal':
> >> {'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', 'MTU': '9000',
> >> 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off',
> >> 'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'}, 'ipv6addrs':
> >> ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000', 'netmask': '', 'stp':
> >> 'off', 'ipv4addrs': [], 'ipv6gateway': '::', 'gateway': '', 'opts':
> >> {'topology_change_detected': '0', 'multicast_last_member_count': '2',
> >> 'hash_elasticity': '4', 'multicast_query_response_interval': '999',
> >> 'multicast_snooping': '1', 'multicast_startup_query_interval': '3124',
> >> 'hello_timer': '31', 'multicast_querier_interval': '25496', 'max_age':
> >> '1999', 'hash_max': '512', 'stp_state': '0', 'root_id':
> >> '8000.001018cddaac', 'priority': '32768', 'multicast_membership_interval':
> >> '25996', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier':
> >> '0',
> >> 'multicast_startup_query_count': '2', 'hello_time': '199',
> >> 'topology_change': '0', 'bridge_id': '8000.001018cddaac',
> >> 'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer': '31',
> >> 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0',
> >> 'multicast_query_interval': '12498', 'multicast_last_member_interval':
> >> '99', 'multicast_router': '1', 'forward_delay': '0'}, 'ports':
> >> ['bond1.100']}, 'VMNetwork': {'addr': '', 'cfg': {'DEFROUTE': 'no',
> >> 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no',
> >> 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'VMNetwork', 'TYPE':
> >> 'Bridge',
> >> 'ONBOOT': 'no'}, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
> >> '1500', 'netmask': '', 'stp': 'off', 'ipv4addrs': [], 'ipv6gateway': '::',
> >> 'gateway': '', 'opts': {'topology_change_detected': '0',
> >> 'multicast_last_member_count': '2', 'hash_elasticity': '4',
> >> 'multicast_query_response_interval': '999', 'multicast_snooping': '1',
> >> 'multicast_startup_query_interval': '3124', 'hello_timer': '131',
> >> 'multicast_querier_interval': '25496', 'max_age': '1999', 'hash_max':
> >> '512', 'stp_state': '0', 'root_id': '8000.60eb6920b46c', 'priority':
> >> '32768', 'multicast_membership_interval': '25996', 'root_path_cost': '0',
> >> 'root_port': '0', 'multicast_querier': '0',
> >> 'multicast_startup_query_count': '2', 'hello_time': '199',
> >> 'topology_change': '0', 'bridge_id': '8000.60eb6920b46c',
> >> 'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer': '31',
> >> 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0',
> >> 'multicast_query_interval': '12498', 'multicast_last_member_interval':
> >> '99', 'multicast_router': '1', 'forward_delay': '0'}, 'ports':
> >> ['bond0.36']}}, 'uuid': '44454C4C-4C00-1057-8053-B7C04F504E31',
> >> 'lastClientIface': 'bond1', 'nics': {'eth3': {'permhwaddr':
> >> '00:10:18:cd:da:ae', 'addr': '', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED':
> >> 'no', 'MTU': '9000', 'HWADDR': '00:10:18:cd:da:ae', 'MASTER': 'bond1',
> >> 'DEVICE': 'eth3', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu': '9000',
> >> 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:10:18:cd:da:ac', 'speed':
> >> 1000}, 'eth2': {'permhwaddr': '00:10:18:cd:da:ac', 'addr': '', 'cfg':
> >> {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR':
> >> '00:10:18:cd:da:ac', 'MASTER': 'bond1', 'DEVICE': 'eth2', 'ONBOOT': 'no'},
> >> 'ipv6addrs': [], 'mtu': '9000', 'netmask': '', 'ipv4addrs': [], 'hwaddr':
> >> '00:10:18:cd:da:ac', 'speed': 1000}, 'eth1': {'permhwaddr':
> >> '60:eb:69:20:b4:6d', 'addr': '', 'cfg': {'SLAVE': 'yes', 'NM_CONTROLLED':
> >> 'no', 'MTU': '1500', 'HWADDR': '60:eb:69:20:b4:6d', 'MASTER': 'bond0',
> >> 'DEVICE': 'eth1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500',
> >> 'netmask': '', 'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'speed':
> >> 1000}, 'eth0': {'permhwaddr': '60:eb:69:20:b4:6c', 'addr': '', 'cfg':
> >> {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR':
> >> '60:eb:69:20:b4:6c', 'MASTER': 'bond0', 'DEVICE': 'eth0', 'ONBOOT':
> >> 'yes'},
> >> 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr':
> >> '60:eb:69:20:b4:6c', 'speed': 1000}}, 'software_revision': '1',
> >> 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'cpuFlags':
> >> u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
> >> 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:32151ce183c8',
> >> 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2',
> >> '3.3',
> >> '3.4', '3.5'], 'autoNumaBalancing': 2, 'reservedMem': '321', 'bondings':
> >> {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves':
> >> [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg':
> >> {'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BONDING_OPTS':
> >> 'mode=4 miimon=100', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'ipv6addrs':
> >> ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500', 'netmask': '',
> >> 'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'slaves': ['eth0',
> >> 'eth1'],
> >> 'opts': {'miimon': '100', 'mode': '4'}}, 'bond1': {'addr': '10.10.10.6',
> >> 'cfg': {'DEFROUTE': 'no', 'IPADDR': '10.10.10.6', 'HOTPLUG': 'no', 'MTU':
> >> '9000', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO':
> >> 'none', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': 'bond1', 'ONBOOT':
> >> 'no'}, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000',
> >> 'netmask': '255.255.255.0', 'ipv4addrs': ['10.10.10.6/24'
> >> <http://10.10.10.6/24%27>], 'hwaddr': '00:10:18:cd:da:ac', 'slaves':
> >> ['eth2', 'eth3'], 'opts': {'miimon': '100', 'mode': '4'}}, 'bond2':
> >> {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [],
> >> 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu':
> >> '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}},
> >> 'software_version': '4.16', 'memSize': '24019', 'cpuSpeed': '2667.000',
> >> 'numaNodes': {u'1': {'totalMemory': '12288', 'cpus': [6, 7, 8, 9, 10, 11,
> >> 18, 19, 20, 21, 22, 23]}, u'0': {'totalMemory': '12278', 'cpus': [0, 1, 2,
> >> 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, 'version_name': 'Snow Man', 'vlans':
> >> {'bond0.10': {'iface': 'bond0', 'addr': '43.25.17.16', 'cfg': {'DEFROUTE':
> >> 'yes', 'VLAN': 'yes', 'IPADDR': '43.25.17.16', 'HOTPLUG': 'no', 'GATEWAY':
> >> '43.25.17.1', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0',
> >> 'BOOTPROTO': 'none', 'DEVICE': 'bond0.10', 'MTU': '1500', 'ONBOOT':
> >> 'yes'},
> >> 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 10, 'mtu':
> >> '1500',
> >> 'netmask': '255.255.255.0', 'ipv4addrs': ['43.25.17.16/24']
> >> <http://43.25.17.16/24%27%5D>}, 'bond0.36': {'iface': 'bond0', 'addr':
> >> '', 'cfg': {'BRIDGE': 'VMNetwork', 'VLAN': 'yes', 'HOTPLUG': 'no', 'MTU':
> >> '1500', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.36', 'ONBOOT': 'no'},
> >> 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 36, 'mtu':
> >> '1500',
> >> 'netmask': '', 'ipv4addrs': []}, 'bond1.100': {'iface': 'bond1', 'addr':
> >> '', 'cfg': {'BRIDGE': 'Internal', 'VLAN': 'yes', 'HOTPLUG': 'no', 'MTU':
> >> '9000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond1.100', 'ONBOOT': 'no'},
> >> 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], 'vlanid': 100, 'mtu':
> >> '9000',
> >> 'netmask': '', 'ipv4addrs': []}}, 'cpuCores': '12', 'kvmEnabled': 'true',
> >> 'guestOverhead': '65', 'cpuThreads': '24', 'emulatedMachines':
> >> [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0',
> >> u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'],
> >> 'operatingSystem': {'release': '5.el6.centos.11.1', 'version': '6',
> >> 'name':
> >> 'RHEL'}, 'lastClient': '10.10.10.2'}}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,620::BindingXMLRPC::1132::vds::(wrapper) client
> >> [10.10.10.2]::call
> >> getHardwareInfo with () {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,621::BindingXMLRPC::1139::vds::(wrapper) return getHardwareInfo
> >> with {'status': {'message': 'Done', 'code': 0}, 'info':
> >> {'systemProductName': 'CS24-TY', 'systemSerialNumber': '7LWSPN1',
> >> 'systemFamily': 'Server', 'systemVersion': 'A00', 'systemUUID':
> >> '44454c4c-4c00-1057-8053-b7c04f504e31', 'systemManufacturer': 'Dell'}}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:41,733::BindingXMLRPC::1132::vds::(wrapper) client
> >> [10.10.10.2]::call
> >> hostsList with () {} flowID [222e8036]
> >> Thread-13::ERROR::2014-11-24
> >> 21:41:44,753::BindingXMLRPC::1148::vds::(wrapper) vdsm exception occured
> >> Traceback (most recent call last):
> >> File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, in wrapper
> >> res = f(*args, **kwargs)
> >> File "/usr/share/vdsm/gluster/api.py", line 54, in wrapper
> >> rv = func(*args, **kwargs)
> >> File "/usr/share/vdsm/gluster/api.py", line 251, in hostsList
> >> return {'hosts': self.svdsmProxy.glusterPeerStatus()}
> >> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> >> return callMethod()
> >> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> >> **kwargs)
> >> File "<string>", line 2, in glusterPeerStatus
> >> File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in
> >> _callmethod
> >> raise convert_to_error(kind, result)
> >> GlusterCmdExecFailedException: Command execution failed
> >> error: Connection failed. Please check if gluster daemon is operational.
> >> return code: 1
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
> >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state init ->
> >> state preparing
> >> Thread-13::INFO::2014-11-24
> >> 21:41:50,950::logUtils::44::dispatcher::(wrapper) Run and protect:
> >> repoStats(options=None)
> >> Thread-13::INFO::2014-11-24
> >> 21:41:50,950::logUtils::47::dispatcher::(wrapper) Run and protect:
> >> repoStats, Return response: {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
> >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
> >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state preparing
> >> ->
> >> state finished
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
> >> Owner.releaseAll requests {} resources {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
> >> Owner.cancelAll requests {}
> >> Thread-13::DEBUG::2014-11-24
> >> 21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
> >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref 0 aborting False
> >> -------------------------------
> >>
> >> [root at compute4 ~]# service glusterd status
> >> glusterd is stopped
> >> [root at compute4 ~]# chkconfig --list | grep glusterd
> >> glusterd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
> >> [root at compute4 ~]#
> >>
> >> Thanks,
> >> Punit
> >>
> >> On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj <kmayilsa at redhat.com> wrote:
> >>
> >>> Can you send the corresponding error in vdsm.log from the host?
> >>>
> >>> Also check if glusterd service is running.
> >>>
> >>> Thanks,
> >>> Kanagaraj
> >>>
> >>>
> >>> On 11/24/2014 03:39 PM, Punit Dambiwal wrote:
> >>>
> >>> Hi,
> >>>
> >>> After reboot my Hypervisior host can not activate again in the cluster
> >>> and failed with the following error :-
> >>>
> >>> Gluster command [<UNKNOWN>] failed on server...
> >>>
> >>> Engine logs :-
> >>>
> >>> 2014-11-24 18:05:28,397 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-64) START,
> >>> GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
> >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90
> >>> 2014-11-24 18:05:30,609 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-64) FINISH, GlusterVolumesListVDSCommand,
> >>> return:
> >>> {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d95203e0},
> >>> log id: 5f251c90
> >>> 2014-11-24 18:05:33,768 INFO
> >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-8)
> >>> [287d570d] Lock Acquired to object EngineLock [exclusiveLocks= key:
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
> >>> , sharedLocks= ]
> >>> 2014-11-24 18:05:33,795 INFO
> >>> [org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] Running command:
> >>> ActivateVdsCommand internal: false. Entities affected : ID:
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDSAction group
> >>> MANIPULATE_HOST
> >>> with role type ADMIN
> >>> 2014-11-24 18:05:33,796 INFO
> >>> [org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] Before acquiring lock in
> >>> order to prevent monitoring for host Compute5 from data-center SV_WTC
> >>> 2014-11-24 18:05:33,797 INFO
> >>> [org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] Lock acquired, from now a
> >>> monitoring of host will be skipped for host Compute5 from data-center
> >>> SV_WTC
> >>> 2014-11-24 18:05:33,817 INFO
> >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] START,
> >>> SetVdsStatusVDSCommand(HostName = Compute5, HostId =
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned,
> >>> nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1cbc7311
> >>> 2014-11-24 18:05:33,820 INFO
> >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] FINISH,
> >>> SetVdsStatusVDSCommand, log id: 1cbc7311
> >>> 2014-11-24 18:05:34,086 INFO
> >>> [org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) Activate finished. Lock released.
> >>> Monitoring can run now for host Compute5 from data-center SV_WTC
> >>> 2014-11-24 18:05:34,088 INFO
> >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (org.ovirt.thread.pool-8-thread-45) Correlation ID: 287d570d, Job ID:
> >>> 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call Stack: null, Custom Event ID:
> >>> -1, Message: Host Compute5 was activated by admin.
> >>> 2014-11-24 18:05:34,090 INFO
> >>> [org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> (org.ovirt.thread.pool-8-thread-45) Lock freed to object EngineLock
> >>> [exclusiveLocks= key: 0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
> >>> , sharedLocks= ]
> >>> 2014-11-24 18:05:35,792 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-55) [3706e836] START,
> >>> GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
> >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832
> >>> 2014-11-24 18:05:37,064 INFO
> >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-69) START,
> >>> GetHardwareInfoVDSCommand(HostName = Compute5, HostId =
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a,
> >>> vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log id:
> >>> 6d560cc2
> >>> 2014-11-24 18:05:37,074 INFO
> >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-69) FINISH, GetHardwareInfoVDSCommand, log
> >>> id: 6d560cc2
> >>> 2014-11-24 18:05:37,093 WARN
> >>> [org.ovirt.engine.core.vdsbroker.VdsManager]
> >>> (DefaultQuartzScheduler_Worker-69) Host Compute5 is running with disabled
> >>> SELinux.
> >>> 2014-11-24 18:05:37,127 INFO
> >>> [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf] Running command:
> >>> HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities
> >>> affected
> >>> : ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
> >>> 2014-11-24 18:05:37,147 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf] START,
> >>> GlusterServersListVDSCommand(HostName = Compute5, HostId =
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87
> >>> 2014-11-24 18:05:37,164 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf] FINISH,
> >>> GlusterServersListVDSCommand, log id: 4faed87
> >>> 2014-11-24 18:05:37,189 INFO
> >>> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Running command:
> >>> SetNonOperationalVdsCommand internal: true. Entities affected : ID:
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
> >>> 2014-11-24 18:05:37,206 INFO
> >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] START,
> >>> SetVdsStatusVDSCommand(HostName = Compute5, HostId =
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=NonOperational,
> >>> nonOperationalReason=GLUSTER_COMMAND_FAILED, stopSpmFailureLogged=false),
> >>> log id: fed5617
> >>> 2014-11-24 18:05:37,209 INFO
> >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] FINISH,
> >>> SetVdsStatusVDSCommand, log id: fed5617
> >>> 2014-11-24 18:05:37,223 ERROR
> >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation ID: 4a84c4e5,
> >>> Job
> >>> ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b, Call Stack: null, Custom Event
> >>> ID: -1, Message: Gluster command [<UNKNOWN>] failed on server Compute5.
> >>> 2014-11-24 18:05:37,243 INFO
> >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Correlation ID: null, Call
> >>> Stack: null, Custom Event ID: -1, Message: Status of host Compute5 was
> >>> set
> >>> to NonOperational.
> >>> 2014-11-24 18:05:37,272 INFO
> >>> [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
> >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Running command:
> >>> HandleVdsVersionCommand internal: true. Entities affected : ID:
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
> >>> 2014-11-24 18:05:37,274 INFO
> >>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host
> >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5 is already in
> >>> NonOperational status for reason GLUSTER_COMMAND_FAILED.
> >>> SetNonOperationalVds command is skipped.
> >>> 2014-11-24 18:05:38,065 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-55) [3706e836] FINISH,
> >>> GlusterVolumesListVDSCommand, return:
> >>> {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4e72a1b1},
> >>> log id: 48a0c832
> >>> 2014-11-24 18:05:43,243 INFO
> >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> (DefaultQuartzScheduler_Worker-35) START,
> >>> GlusterVolumesListVDSCommand(HostName = Compute4, HostId =
> >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc
> >>> ^C
> >>> [root at ccr01 ~]#
> >>>
> >>> Thanks,
> >>> Punit
> >>>
> >>>
> >>> _______________________________________________
> >>> Users mailing
> >>> listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: gluster_after-reboot.txt
URL: <http://lists.ovirt.org/pipermail/users/attachments/20141125/15520634/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Gluster_after_manual_service_restart.txt
URL: <http://lists.ovirt.org/pipermail/users/attachments/20141125/15520634/attachment-0005.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vdsm_aftermanual_restart.txt
URL: <http://lists.ovirt.org/pipermail/users/attachments/20141125/15520634/attachment-0006.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vdsm_just after_reboot.txt
URL: <http://lists.ovirt.org/pipermail/users/attachments/20141125/15520634/attachment-0007.txt>
More information about the Users
mailing list