Re: [ovirt-users] [Gluster-users] Gluster command [<UNKNOWN>] failed on server...

Hi Kaushal, This is the host...which i rebooted...would you mind to let me know how i can make the glusterd sevice come up after network...i am using centos7...if network is the issue... On Wed, Dec 3, 2014 at 11:54 AM, Kaushal M <kshlmster(a)gmail.com&gt; wrote: > > This peer cannot be identified. > > " [2014-12-03 02:29:25.998153] D > [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: > Unable to find friend: cpu05.zne01.hkg1.ovt.36stack.com" > > I don't know why this address is not being resolved during boot time. If > this is a valid peer, the the only reason I can think of this that the > network is not up. > > If you had previously detached the peer forcefully, the that could have > left stale entries in some volumes. In this case as well, GlusterD will fail > to identify the peer. > > Do either of these reasons seem a possibility to you? > > On Dec 3, 2014 8:07 AM, "Punit Dambiwal" <hypunit(a)gmail.com&gt; wrote: >> >> Hi Kaushal, >> >> Please find the logs here :- http://ur1.ca/iyoe5 and http://ur1.ca/iyoed >> >> On Tue, Dec 2, 2014 at 10:43 PM, Kaushal M <kshlmster(a)gmail.com&gt; wrote: >>> >>> Hey Punit, >>> In the logs you've provided, GlusterD appears to be running correctly. >>> Could you provide the logs for the time period when GlusterD attempts to >>> start but fails. >>> >>> ~kaushal >>> >>> On Dec 2, 2014 8:03 PM, "Punit Dambiwal" <hypunit(a)gmail.com&gt; wrote: >>>> >>>> Hi Kaushal, >>>> >>>> Please find the logs here :- http://ur1.ca/iyhs5 and >>>> http://ur1.ca/iyhue >>>> >>>> Thanks, >>>> punit >>>> >>>> >>>> On Tue, Dec 2, 2014 at 12:00 PM, Kaushal M <kshlmster(a)gmail.com&gt; wrote: >>>>> >>>>> Hey Punit, >>>>> Could you start Glusterd in debug mode and provide the logs here? >>>>> To start it in debug mode, append '-LDEBUG' to the ExecStart line in >>>>> the service file. >>>>> >>>>> ~kaushal >>>>> >>>>> On Mon, Dec 1, 2014 at 9:05 AM, Punit Dambiwal <hypunit(a)gmail.com&gt; >>>>> wrote: >>>>> > Hi, >>>>> > >>>>> > Can Any body help me on this ?? >>>>> > >>>>> > On Thu, Nov 27, 2014 at 9:29 AM, Punit Dambiwal <hypunit(a)gmail.com&gt; >>>>> > wrote: >>>>> >> >>>>> >> Hi Kaushal, >>>>> >> >>>>> >> Thanks for the detailed reply....let me explain my setup first :- >>>>> >> >>>>> >> 1. Ovirt Engine >>>>> >> 2. 4* host as well as storage machine (Host and gluster combined) >>>>> >> 3. Every host has 24 bricks... >>>>> >> >>>>> >> Now whenever the host machine reboot...it can come up but can not >>>>> >> join the >>>>> >> cluster again and through the following error "Gluster command >>>>> >> [<UNKNOWN>] >>>>> >> failed on server.." >>>>> >> >>>>> >> Please check my comment in line :- >>>>> >> >>>>> >> 1. Use the same string for doing the peer probe and for the brick >>>>> >> address >>>>> >> during volume create/add-brick. Ideally, we suggest you use >>>>> >> properly >>>>> >> resolvable FQDNs everywhere. If that is not possible, then use only >>>>> >> IP >>>>> >> addresses. Try to avoid short names. >>>>> >> --------------- >>>>> >> [root@cpu05 ~]# gluster peer status >>>>> >> Number of Peers: 3 >>>>> >> >>>>> >> Hostname: cpu03.stack.com >>>>> >> Uuid: 5729b8c4-e80d-4353-b456-6f467bddbdfb >>>>> >> State: Peer in Cluster (Connected) >>>>> >> >>>>> >> Hostname: cpu04.stack.com >>>>> >> Uuid: d272b790-c4b2-4bed-ba68-793656e6d7b0 >>>>> >> State: Peer in Cluster (Connected) >>>>> >> Other names: >>>>> >> 10.10.0.8 >>>>> >> >>>>> >> Hostname: cpu02.stack.com >>>>> >> Uuid: 8d8a7041-950e-40d0-85f9-58d14340ca25 >>>>> >> State: Peer in Cluster (Connected) >>>>> >> [root@cpu05 ~]# >>>>> >> ---------------- >>>>> >> 2. During boot up, make sure to launch glusterd only after the >>>>> >> network is >>>>> >> up. This will allow the new peer identification mechanism to do its >>>>> >> job correctly. >>>>> >> >> I think the service itself doing the same job.... >>>>> >> >>>>> >> [root@cpu05 ~]# cat /usr/lib/systemd/system/glusterd.service >>>>> >> [Unit] >>>>> >> Description=GlusterFS, a clustered file-system server >>>>> >> After=network.target rpcbind.service >>>>> >> Before=network-online.target >>>>> >> >>>>> >> [Service] >>>>> >> Type=forking >>>>> >> PIDFile=/var/run/glusterd.pid >>>>> >> LimitNOFILE=65536 >>>>> >> ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid >>>>> >> KillMode=process >>>>> >> >>>>> >> [Install] >>>>> >> WantedBy=multi-user.target >>>>> >> [root@cpu05 ~]# >>>>> >> -------------------- >>>>> >> >>>>> >> gluster logs :- >>>>> >> >>>>> >> [2014-11-24 09:22:22.147471] I [MSGID: 100030] >>>>> >> [glusterfsd.c:2018:main] >>>>> >> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version >>>>> >> 3.6.1 >>>>> >> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) >>>>> >> [2014-11-24 09:22:22.151565] I [glusterd.c:1214:init] 0-management: >>>>> >> Maximum allowed open file descriptors set to 65536 >>>>> >> [2014-11-24 09:22:22.151599] I [glusterd.c:1259:init] 0-management: >>>>> >> Using >>>>> >> /var/lib/glusterd as working directory >>>>> >> [2014-11-24 09:22:22.155216] W [rdma.c:4195:__gf_rdma_ctx_create] >>>>> >> 0-rpc-transport/rdma: rdma_cm event channel creation failed (No >>>>> >> such device) >>>>> >> [2014-11-24 09:22:22.155264] E [rdma.c:4483:init] >>>>> >> 0-rdma.management: >>>>> >> Failed to initialize IB Device >>>>> >> [2014-11-24 09:22:22.155285] E >>>>> >> [rpc-transport.c:333:rpc_transport_load] >>>>> >> 0-rpc-transport: 'rdma' initialization failed >>>>> >> [2014-11-24 09:22:22.155354] W >>>>> >> [rpcsvc.c:1524:rpcsvc_transport_create] >>>>> >> 0-rpc-service: cannot create listener, initing the transport failed >>>>> >> [2014-11-24 09:22:22.156290] I >>>>> >> [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: >>>>> >> geo-replication >>>>> >> module not installed in the system >>>>> >> [2014-11-24 09:22:22.161318] I >>>>> >> [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: >>>>> >> retrieved >>>>> >> op-version: 30600 >>>>> >> [2014-11-24 09:22:22.821800] I >>>>> >> [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] >>>>> >> 0-management: >>>>> >> connect returned 0 >>>>> >> [2014-11-24 09:22:22.825810] I >>>>> >> [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] >>>>> >> 0-management: >>>>> >> connect returned 0 >>>>> >> [2014-11-24 09:22:22.828705] I >>>>> >> [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] >>>>> >> 0-management: >>>>> >> connect returned 0 >>>>> >> [2014-11-24 09:22:22.828771] I >>>>> >> [rpc-clnt.c:969:rpc_clnt_connection_init] >>>>> >> 0-management: setting frame-timeout to 600 >>>>> >> [2014-11-24 09:22:22.832670] I >>>>> >> [rpc-clnt.c:969:rpc_clnt_connection_init] >>>>> >> 0-management: setting frame-timeout to 600 >>>>> >> [2014-11-24 09:22:22.835919] I >>>>> >> [rpc-clnt.c:969:rpc_clnt_connection_init] >>>>> >> 0-management: setting frame-timeout to 600 >>>>> >> [2014-11-24 09:22:22.840209] E >>>>> >> [glusterd-store.c:4248:glusterd_resolve_all_bricks] 0-glusterd: >>>>> >> resolve >>>>> >> brick failed in restore >>>>> >> [2014-11-24 09:22:22.840233] E [xlator.c:425:xlator_init] >>>>> >> 0-management: >>>>> >> Initialization of volume 'management' failed, review your volfile >>>>> >> again >>>>> >> [2014-11-24 09:22:22.840245] E [graph.c:322:glusterfs_graph_init] >>>>> >> 0-management: initializing translator failed >>>>> >> [2014-11-24 09:22:22.840264] E >>>>> >> [graph.c:525:glusterfs_graph_activate] >>>>> >> 0-graph: init failed >>>>> >> [2014-11-24 09:22:22.840754] W [glusterfsd.c:1194:cleanup_and_exit] >>>>> >> (--> >>>>> >> 0-: received signum (0), shutting down >>>>> >> >>>>> >> Thanks, >>>>> >> Punit >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Wed, Nov 26, 2014 at 7:14 PM, Kaushal M <kshlmster(a)gmail.com&gt; >>>>> >> wrote: >>>>> >>> >>>>> >>> Based on the logs I can guess that glusterd is being started >>>>> >>> before >>>>> >>> the network has come up and that the addresses given to bricks do >>>>> >>> not >>>>> >>> directly match the addresses used in during peer probe. >>>>> >>> >>>>> >>> The gluster_after_reboot log has the line "[2014-11-25 >>>>> >>> 06:46:09.972113] E >>>>> >>> [glusterd-store.c:2632:glusterd_resolve_all_bricks] >>>>> >>> 0-glusterd: resolve brick failed in restore". >>>>> >>> >>>>> >>> Brick resolution fails when glusterd cannot match the address for >>>>> >>> the >>>>> >>> brick, with one of the peers. Brick resolution happens in two >>>>> >>> phases, >>>>> >>> 1. We first try to identify the peer by performing string >>>>> >>> comparisions >>>>> >>> with the brick address and the peer addresses (The peer names will >>>>> >>> be >>>>> >>> the names/addresses that were given when the peer was probed). >>>>> >>> 2. If we don't find a match from step 1, we will then resolve all >>>>> >>> the >>>>> >>> brick address and the peer addresses into addrinfo structs, and >>>>> >>> then >>>>> >>> compare these structs to find a match. This process should >>>>> >>> generally >>>>> >>> find a match if available. This will fail only if the network is >>>>> >>> not >>>>> >>> up yet as we cannot resolve addresses. >>>>> >>> >>>>> >>> The above steps are applicable only to glusterfs versions >=3.6. >>>>> >>> They >>>>> >>> were introduced to reduce problems with peer identification, like >>>>> >>> the >>>>> >>> one you encountered >>>>> >>> >>>>> >>> Since both of the steps failed to find a match in one run, but >>>>> >>> succeeded later, we can come to the conclusion that, >>>>> >>> a) the bricks don't have the exact same string used in peer probe >>>>> >>> for >>>>> >>> their addresses as step 1 failed, and >>>>> >>> b) the network was not up in the initial run, as step 2 failed >>>>> >>> during >>>>> >>> the initial run, but passed in the second run. >>>>> >>> >>>>> >>> Please let me know if my conclusion is correct. >>>>> >>> >>>>> >>> If it is, you can solve your problem in two ways. >>>>> >>> 1. Use the same string for doing the peer probe and for the brick >>>>> >>> address during volume create/add-brick. Ideally, we suggest you >>>>> >>> use >>>>> >>> properly resolvable FQDNs everywhere. If that is not possible, >>>>> >>> then >>>>> >>> use only IP addresses. Try to avoid short names. >>>>> >>> 2. During boot up, make sure to launch glusterd only after the >>>>> >>> network >>>>> >>> is up. This will allow the new peer identification mechanism to do >>>>> >>> its >>>>> >>> job correctly. >>>>> >>> >>>>> >>> >>>>> >>> If you have already followed these steps and yet still hit the >>>>> >>> problem, then please provide more information (setup, logs, etc.). >>>>> >>> It >>>>> >>> could be much different problem that you are facing. >>>>> >>> >>>>> >>> ~kaushal >>>>> >>> >>>>> >>> On Wed, Nov 26, 2014 at 4:01 PM, Punit Dambiwal >>>>> >>> <hypunit(a)gmail.com&gt; >>>>> >>> wrote: >>>>> >>> > Is there any one can help on this ?? >>>>> >>> > >>>>> >>> > Thanks, >>>>> >>> > punit >>>>> >>> > >>>>> >>> > On Wed, Nov 26, 2014 at 9:42 AM, Punit Dambiwal >>>>> >>> > <hypunit(a)gmail.com&gt; >>>>> >>> > wrote: >>>>> >>> >> >>>>> >>> >> Hi, >>>>> >>> >> >>>>> >>> >> My Glusterfs version is :- glusterfs-3.6.1-1.el7 >>>>> >>> >> >>>>> >>> >> On Wed, Nov 26, 2014 at 1:59 AM, Kanagaraj Mayilsamy >>>>> >>> >> <kmayilsa(a)redhat.com&gt; >>>>> >>> >> wrote: >>>>> >>> >>> >>>>> >>> >>> [+Gluster-users(a)gluster.org] >>>>> >>> >>> >>>>> >>> >>> "Initialization of volume 'management' failed, review your >>>>> >>> >>> volfile >>>>> >>> >>> again", glusterd throws this error when the service is started >>>>> >>> >>> automatically >>>>> >>> >>> after the reboot. But the service is successfully started >>>>> >>> >>> later >>>>> >>> >>> manually by >>>>> >>> >>> the user. >>>>> >>> >>> >>>>> >>> >>> can somebody from gluster-users please help on this? >>>>> >>> >>> >>>>> >>> >>> glusterfs version: 3.5.1 >>>>> >>> >>> >>>>> >>> >>> Thanks, >>>>> >>> >>> Kanagaraj >>>>> >>> >>> >>>>> >>> >>> ----- Original Message ----- >>>>> >>> >>> > From: "Punit Dambiwal" <hypunit(a)gmail.com&gt; >>>>> >>> >>> > To: "Kanagaraj" <kmayilsa(a)redhat.com&gt; >>>>> >>> >>> > Cc: users(a)ovirt.org >>>>> >>> >>> > Sent: Tuesday, November 25, 2014 7:24:45 PM >>>>> >>> >>> > Subject: Re: [ovirt-users] Gluster command [<UNKNOWN>] >>>>> >>> >>> > failed on >>>>> >>> >>> > server... >>>>> >>> >>> > >>>>> >>> >>> > Hi Kanagraj, >>>>> >>> >>> > >>>>> >>> >>> > Please check the attached log files....i didn't find any >>>>> >>> >>> > thing >>>>> >>> >>> > special.... >>>>> >>> >>> > >>>>> >>> >>> > On Tue, Nov 25, 2014 at 12:12 PM, Kanagaraj >>>>> >>> >>> > <kmayilsa(a)redhat.com&gt; >>>>> >>> >>> > wrote: >>>>> >>> >>> > >>>>> >>> >>> > > Do you see any errors in >>>>> >>> >>> > > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log or >>>>> >>> >>> > > vdsm.log >>>>> >>> >>> > > when >>>>> >>> >>> > > the >>>>> >>> >>> > > service is trying to start automatically after the reboot? >>>>> >>> >>> > > >>>>> >>> >>> > > Thanks, >>>>> >>> >>> > > Kanagaraj >>>>> >>> >>> > > >>>>> >>> >>> > > >>>>> >>> >>> > > On 11/24/2014 08:13 PM, Punit Dambiwal wrote: >>>>> >>> >>> > > >>>>> >>> >>> > > Hi Kanagaraj, >>>>> >>> >>> > > >>>>> >>> >>> > > Yes...once i will start the gluster service and then >>>>> >>> >>> > > vdsmd >>>>> >>> >>> > > ...the >>>>> >>> >>> > > host >>>>> >>> >>> > > can connect to cluster...but the question is why it's not >>>>> >>> >>> > > started >>>>> >>> >>> > > even it >>>>> >>> >>> > > has chkconfig enabled... >>>>> >>> >>> > > >>>>> >>> >>> > > I have tested it in two host cluster >>>>> >>> >>> > > environment...(Centos 6.6 >>>>> >>> >>> > > and >>>>> >>> >>> > > centos 7.0) on both hypervisior cluster..it's failed to >>>>> >>> >>> > > reconnect >>>>> >>> >>> > > in >>>>> >>> >>> > > to >>>>> >>> >>> > > cluster after reboot.... >>>>> >>> >>> > > >>>>> >>> >>> > > In both the environment glusterd enabled for next >>>>> >>> >>> > > boot....but >>>>> >>> >>> > > it's >>>>> >>> >>> > > failed with the same error....seems it's bug in either >>>>> >>> >>> > > gluster or >>>>> >>> >>> > > Ovirt ?? >>>>> >>> >>> > > >>>>> >>> >>> > > Please help me to find the workaround here if can not >>>>> >>> >>> > > resolve >>>>> >>> >>> > > it...as >>>>> >>> >>> > > without this the Host machine can not connect after >>>>> >>> >>> > > reboot....that >>>>> >>> >>> > > means >>>>> >>> >>> > > engine will consider it as down and every time need to >>>>> >>> >>> > > manually >>>>> >>> >>> > > start >>>>> >>> >>> > > the >>>>> >>> >>> > > gluster service and vdsmd... ?? >>>>> >>> >>> > > >>>>> >>> >>> > > Thanks, >>>>> >>> >>> > > Punit >>>>> >>> >>> > > >>>>> >>> >>> > > On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj >>>>> >>> >>> > > <kmayilsa(a)redhat.com&gt; >>>>> >>> >>> > > wrote: >>>>> >>> >>> > > >>>>> >>> >>> > >> From vdsm.log "error: Connection failed. Please check if >>>>> >>> >>> > >> gluster >>>>> >>> >>> > >> daemon >>>>> >>> >>> > >> is operational." >>>>> >>> >>> > >> >>>>> >>> >>> > >> Starting glusterd service should fix this issue. 'service >>>>> >>> >>> > >> glusterd >>>>> >>> >>> > >> start' >>>>> >>> >>> > >> But i am wondering why the glusterd was not started >>>>> >>> >>> > >> automatically >>>>> >>> >>> > >> after >>>>> >>> >>> > >> the reboot. >>>>> >>> >>> > >> >>>>> >>> >>> > >> Thanks, >>>>> >>> >>> > >> Kanagaraj >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> On 11/24/2014 07:18 PM, Punit Dambiwal wrote: >>>>> >>> >>> > >> >>>>> >>> >>> > >> Hi Kanagaraj, >>>>> >>> >>> > >> >>>>> >>> >>> > >> Please find the attached VDSM logs :- >>>>> >>> >>> > >> >>>>> >>> >>> > >> ---------------- >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) >>>>> >>> >>> > >> Owner.cancelAll requests {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:17,182::task::993::Storage.TaskManager.Task::(_decref) >>>>> >>> >>> > >> Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref 0 >>>>> >>> >>> > >> aborting >>>>> >>> >>> > >> False >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState) >>>>> >>> >>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from >>>>> >>> >>> > >> state >>>>> >>> >>> > >> init >>>>> >>> >>> > >> -> >>>>> >>> >>> > >> state preparing >>>>> >>> >>> > >> Thread-13::INFO::2014-11-24 >>>>> >>> >>> > >> 21:41:32,393::logUtils::44::dispatcher::(wrapper) Run and >>>>> >>> >>> > >> protect: >>>>> >>> >>> > >> repoStats(options=None) >>>>> >>> >>> > >> Thread-13::INFO::2014-11-24 >>>>> >>> >>> > >> 21:41:32,393::logUtils::47::dispatcher::(wrapper) Run and >>>>> >>> >>> > >> protect: >>>>> >>> >>> > >> repoStats, Return response: {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare) >>>>> >>> >>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState) >>>>> >>> >>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from >>>>> >>> >>> > >> state >>>>> >>> >>> > >> preparing >>>>> >>> >>> > >> -> >>>>> >>> >>> > >> state finished >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) >>>>> >>> >>> > >> Owner.releaseAll requests {} resources {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) >>>>> >>> >>> > >> Owner.cancelAll requests {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:32,394::task::993::Storage.TaskManager.Task::(_decref) >>>>> >>> >>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref 0 >>>>> >>> >>> > >> aborting >>>>> >>> >>> > >> False >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,550::BindingXMLRPC::1132::vds::(wrapper) client >>>>> >>> >>> > >> [10.10.10.2]::call >>>>> >>> >>> > >> getCapabilities with () {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,553::utils::738::root::(execCmd) >>>>> >>> >>> > >> /sbin/ip route show to 0.0.0.0/0 table all (cwd None) >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,560::utils::758::root::(execCmd) >>>>> >>> >>> > >> SUCCESS: <err> = ''; <rc> = 0 >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,588::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,592::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift-object',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,593::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift-plugin',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift-account',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift-proxy',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift-doc',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,599::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('gluster-swift-container',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,599::caps::728::root::(_getKeyPackages) rpm >>>>> >>> >>> > >> package >>>>> >>> >>> > >> ('glusterfs-geo-replication',) not found >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,600::caps::646::root::(get) >>>>> >>> >>> > >> VirtioRNG DISABLED: libvirt version 0.10.2-29.el6_5.9 >>>>> >>> >>> > >> required >>>>> >>> >>> > >> >= >>>>> >>> >>> > >> 0.10.2-31 >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,603::BindingXMLRPC::1139::vds::(wrapper) return >>>>> >>> >>> > >> getCapabilities >>>>> >>> >>> > >> with {'status': {'message': 'Done', 'code': 0}, 'info': >>>>> >>> >>> > >> {'HBAInventory': >>>>> >>> >>> > >> {'iSCSI': [{'InitiatorName': >>>>> >>> >>> > >> 'iqn.1994-05.com.redhat:32151ce183c8'}], >>>>> >>> >>> > >> 'FC': >>>>> >>> >>> > >> []}, 'packages2': {'kernel': {'release': >>>>> >>> >>> > >> '431.el6.x86_64', >>>>> >>> >>> > >> 'buildtime': >>>>> >>> >>> > >> 1385061309.0, 'version': '2.6.32'}, 'glusterfs-rdma': >>>>> >>> >>> > >> {'release': >>>>> >>> >>> > >> '1.el6', >>>>> >>> >>> > >> 'buildtime': 1403622628L, 'version': '3.5.1'}, >>>>> >>> >>> > >> 'glusterfs-fuse': >>>>> >>> >>> > >> {'release': '1.el6', 'buildtime': 1403622628L, 'version': >>>>> >>> >>> > >> '3.5.1'}, >>>>> >>> >>> > >> 'spice-server': {'release': '6.el6_5.2', 'buildtime': >>>>> >>> >>> > >> 1402324637L, >>>>> >>> >>> > >> 'version': '0.12.4'}, 'vdsm': {'release': >>>>> >>> >>> > >> '1.gitdb83943.el6', >>>>> >>> >>> > >> 'buildtime': >>>>> >>> >>> > >> 1412784567L, 'version': '4.16.7'}, 'qemu-kvm': >>>>> >>> >>> > >> {'release': >>>>> >>> >>> > >> '2.415.el6_5.10', 'buildtime': 1402435700L, 'version': >>>>> >>> >>> > >> '0.12.1.2'}, >>>>> >>> >>> > >> 'qemu-img': {'release': '2.415.el6_5.10', 'buildtime': >>>>> >>> >>> > >> 1402435700L, >>>>> >>> >>> > >> 'version': '0.12.1.2'}, 'libvirt': {'release': >>>>> >>> >>> > >> '29.el6_5.9', >>>>> >>> >>> > >> 'buildtime': >>>>> >>> >>> > >> 1402404612L, 'version': '0.10.2'}, 'glusterfs': >>>>> >>> >>> > >> {'release': >>>>> >>> >>> > >> '1.el6', >>>>> >>> >>> > >> 'buildtime': 1403622628L, 'version': '3.5.1'}, 'mom': >>>>> >>> >>> > >> {'release': >>>>> >>> >>> > >> '2.el6', >>>>> >>> >>> > >> 'buildtime': 1403794344L, 'version': '0.4.1'}, >>>>> >>> >>> > >> 'glusterfs-server': >>>>> >>> >>> > >> {'release': '1.el6', 'buildtime': 1403622628L, 'version': >>>>> >>> >>> > >> '3.5.1'}}, >>>>> >>> >>> > >> 'numaNodeDistance': {'1': [20, 10], '0': [10, 20]}, >>>>> >>> >>> > >> 'cpuModel': >>>>> >>> >>> > >> 'Intel(R) >>>>> >>> >>> > >> Xeon(R) CPU X5650 @ 2.67GHz', 'liveMerge': >>>>> >>> >>> > >> 'false', >>>>> >>> >>> > >> 'hooks': >>>>> >>> >>> > >> {}, >>>>> >>> >>> > >> 'cpuSockets': '2', 'vmTypes': ['kvm'], 'selinux': >>>>> >>> >>> > >> {'mode': '1'}, >>>>> >>> >>> > >> 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], >>>>> >>> >>> > >> 'networks': >>>>> >>> >>> > >> {'ovirtmgmt': {'iface': u'bond0.10', 'addr': >>>>> >>> >>> > >> '43.252.176.16', >>>>> >>> >>> > >> 'bridged': >>>>> >>> >>> > >> False, 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', >>>>> >>> >>> > >> 'bootproto4': 'none', 'netmask': '255.255.255.0', >>>>> >>> >>> > >> 'ipv4addrs': >>>>> >>> >>> > >> [' >>>>> >>> >>> > >> 43.252.176.16/24' <http://43.252.176.16/24%27>], >>>>> >>> >>> > >> 'interface': >>>>> >>> >>> > >> u'bond0.10', 'ipv6gateway': '::', 'gateway': >>>>> >>> >>> > >> '43.25.17.1'}, >>>>> >>> >>> > >> 'Internal': >>>>> >>> >>> > >> {'iface': 'Internal', 'addr': '', 'cfg': {'DEFROUTE': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'HOTPLUG': >>>>> >>> >>> > >> 'no', 'MTU': '9000', 'DELAY': '0', 'NM_CONTROLLED': 'no', >>>>> >>> >>> > >> 'BOOTPROTO': >>>>> >>> >>> > >> 'none', 'STP': 'off', 'DEVICE': 'Internal', 'TYPE': >>>>> >>> >>> > >> 'Bridge', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'no'}, 'bridged': True, 'ipv6addrs': >>>>> >>> >>> > >> ['fe80::210:18ff:fecd:daac/64'], >>>>> >>> >>> > >> 'gateway': '', 'bootproto4': 'none', 'netmask': '', >>>>> >>> >>> > >> 'stp': >>>>> >>> >>> > >> 'off', >>>>> >>> >>> > >> 'ipv4addrs': [], 'mtu': '9000', 'ipv6gateway': '::', >>>>> >>> >>> > >> 'ports': >>>>> >>> >>> > >> ['bond1.100']}, 'storage': {'iface': u'bond1', 'addr': >>>>> >>> >>> > >> '10.10.10.6', >>>>> >>> >>> > >> 'bridged': False, 'ipv6addrs': >>>>> >>> >>> > >> ['fe80::210:18ff:fecd:daac/64'], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '9000', 'bootproto4': 'none', 'netmask': '255.255.255.0', >>>>> >>> >>> > >> 'ipv4addrs': [' >>>>> >>> >>> > >> 10.10.10.6/24' <http://10.10.10.6/24%27>], 'interface': >>>>> >>> >>> > >> u'bond1', >>>>> >>> >>> > >> 'ipv6gateway': '::', 'gateway': ''}, 'VMNetwork': >>>>> >>> >>> > >> {'iface': >>>>> >>> >>> > >> 'VMNetwork', >>>>> >>> >>> > >> 'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', >>>>> >>> >>> > >> 'MTU': >>>>> >>> >>> > >> '1500', >>>>> >>> >>> > >> 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', >>>>> >>> >>> > >> 'STP': >>>>> >>> >>> > >> 'off', >>>>> >>> >>> > >> 'DEVICE': 'VMNetwork', 'TYPE': 'Bridge', 'ONBOOT': 'no'}, >>>>> >>> >>> > >> 'bridged': >>>>> >>> >>> > >> True, >>>>> >>> >>> > >> 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'gateway': >>>>> >>> >>> > >> '', >>>>> >>> >>> > >> 'bootproto4': >>>>> >>> >>> > >> 'none', 'netmask': '', 'stp': 'off', 'ipv4addrs': [], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', >>>>> >>> >>> > >> 'ipv6gateway': '::', 'ports': ['bond0.36']}}, 'bridges': >>>>> >>> >>> > >> {'Internal': >>>>> >>> >>> > >> {'addr': '', 'cfg': {'DEFROUTE': 'no', 'HOTPLUG': 'no', >>>>> >>> >>> > >> 'MTU': >>>>> >>> >>> > >> '9000', >>>>> >>> >>> > >> 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', >>>>> >>> >>> > >> 'STP': >>>>> >>> >>> > >> 'off', >>>>> >>> >>> > >> 'DEVICE': 'Internal', 'TYPE': 'Bridge', 'ONBOOT': 'no'}, >>>>> >>> >>> > >> 'ipv6addrs': >>>>> >>> >>> > >> ['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000', >>>>> >>> >>> > >> 'netmask': '', >>>>> >>> >>> > >> 'stp': >>>>> >>> >>> > >> 'off', 'ipv4addrs': [], 'ipv6gateway': '::', 'gateway': >>>>> >>> >>> > >> '', >>>>> >>> >>> > >> 'opts': >>>>> >>> >>> > >> {'topology_change_detected': '0', >>>>> >>> >>> > >> 'multicast_last_member_count': >>>>> >>> >>> > >> '2', >>>>> >>> >>> > >> 'hash_elasticity': '4', >>>>> >>> >>> > >> 'multicast_query_response_interval': >>>>> >>> >>> > >> '999', >>>>> >>> >>> > >> 'multicast_snooping': '1', >>>>> >>> >>> > >> 'multicast_startup_query_interval': >>>>> >>> >>> > >> '3124', >>>>> >>> >>> > >> 'hello_timer': '31', 'multicast_querier_interval': >>>>> >>> >>> > >> '25496', >>>>> >>> >>> > >> 'max_age': >>>>> >>> >>> > >> '1999', 'hash_max': '512', 'stp_state': '0', 'root_id': >>>>> >>> >>> > >> '8000.001018cddaac', 'priority': '32768', >>>>> >>> >>> > >> 'multicast_membership_interval': >>>>> >>> >>> > >> '25996', 'root_path_cost': '0', 'root_port': '0', >>>>> >>> >>> > >> 'multicast_querier': >>>>> >>> >>> > >> '0', >>>>> >>> >>> > >> 'multicast_startup_query_count': '2', 'hello_time': >>>>> >>> >>> > >> '199', >>>>> >>> >>> > >> 'topology_change': '0', 'bridge_id': '8000.001018cddaac', >>>>> >>> >>> > >> 'topology_change_timer': '0', 'ageing_time': '29995', >>>>> >>> >>> > >> 'gc_timer': >>>>> >>> >>> > >> '31', >>>>> >>> >>> > >> 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0', >>>>> >>> >>> > >> 'multicast_query_interval': '12498', >>>>> >>> >>> > >> 'multicast_last_member_interval': >>>>> >>> >>> > >> '99', 'multicast_router': '1', 'forward_delay': '0'}, >>>>> >>> >>> > >> 'ports': >>>>> >>> >>> > >> ['bond1.100']}, 'VMNetwork': {'addr': '', 'cfg': >>>>> >>> >>> > >> {'DEFROUTE': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', >>>>> >>> >>> > >> 'NM_CONTROLLED': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'VMNetwork', >>>>> >>> >>> > >> 'TYPE': >>>>> >>> >>> > >> 'Bridge', >>>>> >>> >>> > >> 'ONBOOT': 'no'}, 'ipv6addrs': >>>>> >>> >>> > >> ['fe80::62eb:69ff:fe20:b46c/64'], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', 'netmask': '', 'stp': 'off', 'ipv4addrs': [], >>>>> >>> >>> > >> 'ipv6gateway': >>>>> >>> >>> > >> '::', >>>>> >>> >>> > >> 'gateway': '', 'opts': {'topology_change_detected': '0', >>>>> >>> >>> > >> 'multicast_last_member_count': '2', 'hash_elasticity': >>>>> >>> >>> > >> '4', >>>>> >>> >>> > >> 'multicast_query_response_interval': '999', >>>>> >>> >>> > >> 'multicast_snooping': >>>>> >>> >>> > >> '1', >>>>> >>> >>> > >> 'multicast_startup_query_interval': '3124', >>>>> >>> >>> > >> 'hello_timer': >>>>> >>> >>> > >> '131', >>>>> >>> >>> > >> 'multicast_querier_interval': '25496', 'max_age': '1999', >>>>> >>> >>> > >> 'hash_max': >>>>> >>> >>> > >> '512', 'stp_state': '0', 'root_id': '8000.60eb6920b46c', >>>>> >>> >>> > >> 'priority': >>>>> >>> >>> > >> '32768', 'multicast_membership_interval': '25996', >>>>> >>> >>> > >> 'root_path_cost': >>>>> >>> >>> > >> '0', >>>>> >>> >>> > >> 'root_port': '0', 'multicast_querier': '0', >>>>> >>> >>> > >> 'multicast_startup_query_count': '2', 'hello_time': >>>>> >>> >>> > >> '199', >>>>> >>> >>> > >> 'topology_change': '0', 'bridge_id': '8000.60eb6920b46c', >>>>> >>> >>> > >> 'topology_change_timer': '0', 'ageing_time': '29995', >>>>> >>> >>> > >> 'gc_timer': >>>>> >>> >>> > >> '31', >>>>> >>> >>> > >> 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0', >>>>> >>> >>> > >> 'multicast_query_interval': '12498', >>>>> >>> >>> > >> 'multicast_last_member_interval': >>>>> >>> >>> > >> '99', 'multicast_router': '1', 'forward_delay': '0'}, >>>>> >>> >>> > >> 'ports': >>>>> >>> >>> > >> ['bond0.36']}}, 'uuid': >>>>> >>> >>> > >> '44454C4C-4C00-1057-8053-B7C04F504E31', >>>>> >>> >>> > >> 'lastClientIface': 'bond1', 'nics': {'eth3': >>>>> >>> >>> > >> {'permhwaddr': >>>>> >>> >>> > >> '00:10:18:cd:da:ae', 'addr': '', 'cfg': {'SLAVE': 'yes', >>>>> >>> >>> > >> 'NM_CONTROLLED': >>>>> >>> >>> > >> 'no', 'MTU': '9000', 'HWADDR': '00:10:18:cd:da:ae', >>>>> >>> >>> > >> 'MASTER': >>>>> >>> >>> > >> 'bond1', >>>>> >>> >>> > >> 'DEVICE': 'eth3', 'ONBOOT': 'no'}, 'ipv6addrs': [], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '9000', >>>>> >>> >>> > >> 'netmask': '', 'ipv4addrs': [], 'hwaddr': >>>>> >>> >>> > >> '00:10:18:cd:da:ac', >>>>> >>> >>> > >> 'speed': >>>>> >>> >>> > >> 1000}, 'eth2': {'permhwaddr': '00:10:18:cd:da:ac', >>>>> >>> >>> > >> 'addr': '', >>>>> >>> >>> > >> 'cfg': >>>>> >>> >>> > >> {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '9000', >>>>> >>> >>> > >> 'HWADDR': >>>>> >>> >>> > >> '00:10:18:cd:da:ac', 'MASTER': 'bond1', 'DEVICE': 'eth2', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'no'}, >>>>> >>> >>> > >> 'ipv6addrs': [], 'mtu': '9000', 'netmask': '', >>>>> >>> >>> > >> 'ipv4addrs': [], >>>>> >>> >>> > >> 'hwaddr': >>>>> >>> >>> > >> '00:10:18:cd:da:ac', 'speed': 1000}, 'eth1': >>>>> >>> >>> > >> {'permhwaddr': >>>>> >>> >>> > >> '60:eb:69:20:b4:6d', 'addr': '', 'cfg': {'SLAVE': 'yes', >>>>> >>> >>> > >> 'NM_CONTROLLED': >>>>> >>> >>> > >> 'no', 'MTU': '1500', 'HWADDR': '60:eb:69:20:b4:6d', >>>>> >>> >>> > >> 'MASTER': >>>>> >>> >>> > >> 'bond0', >>>>> >>> >>> > >> 'DEVICE': 'eth1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', >>>>> >>> >>> > >> 'netmask': '', 'ipv4addrs': [], 'hwaddr': >>>>> >>> >>> > >> '60:eb:69:20:b4:6c', >>>>> >>> >>> > >> 'speed': >>>>> >>> >>> > >> 1000}, 'eth0': {'permhwaddr': '60:eb:69:20:b4:6c', >>>>> >>> >>> > >> 'addr': '', >>>>> >>> >>> > >> 'cfg': >>>>> >>> >>> > >> {'SLAVE': 'yes', 'NM_CONTROLLED': 'no', 'MTU': '1500', >>>>> >>> >>> > >> 'HWADDR': >>>>> >>> >>> > >> '60:eb:69:20:b4:6c', 'MASTER': 'bond0', 'DEVICE': 'eth0', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'yes'}, >>>>> >>> >>> > >> 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', >>>>> >>> >>> > >> 'ipv4addrs': [], >>>>> >>> >>> > >> 'hwaddr': >>>>> >>> >>> > >> '60:eb:69:20:b4:6c', 'speed': 1000}}, >>>>> >>> >>> > >> 'software_revision': '1', >>>>> >>> >>> > >> 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4', >>>>> >>> >>> > >> '3.5'], >>>>> >>> >>> > >> 'cpuFlags': >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270', >>>>> >>> >>> > >> 'ISCSIInitiatorName': >>>>> >>> >>> > >> 'iqn.1994-05.com.redhat:32151ce183c8', >>>>> >>> >>> > >> 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', >>>>> >>> >>> > >> '3.1', >>>>> >>> >>> > >> '3.2', >>>>> >>> >>> > >> '3.3', >>>>> >>> >>> > >> '3.4', '3.5'], 'autoNumaBalancing': 2, 'reservedMem': >>>>> >>> >>> > >> '321', >>>>> >>> >>> > >> 'bondings': >>>>> >>> >>> > >> {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', >>>>> >>> >>> > >> 'netmask': '', >>>>> >>> >>> > >> 'slaves': >>>>> >>> >>> > >> [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', >>>>> >>> >>> > >> 'cfg': >>>>> >>> >>> > >> {'HOTPLUG': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', >>>>> >>> >>> > >> 'BONDING_OPTS': >>>>> >>> >>> > >> 'mode=4 miimon=100', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, >>>>> >>> >>> > >> 'ipv6addrs': >>>>> >>> >>> > >> ['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500', >>>>> >>> >>> > >> 'netmask': '', >>>>> >>> >>> > >> 'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c', 'slaves': >>>>> >>> >>> > >> ['eth0', >>>>> >>> >>> > >> 'eth1'], >>>>> >>> >>> > >> 'opts': {'miimon': '100', 'mode': '4'}}, 'bond1': >>>>> >>> >>> > >> {'addr': >>>>> >>> >>> > >> '10.10.10.6', >>>>> >>> >>> > >> 'cfg': {'DEFROUTE': 'no', 'IPADDR': '10.10.10.6', >>>>> >>> >>> > >> 'HOTPLUG': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'MTU': >>>>> >>> >>> > >> '9000', 'NM_CONTROLLED': 'no', 'NETMASK': >>>>> >>> >>> > >> '255.255.255.0', >>>>> >>> >>> > >> 'BOOTPROTO': >>>>> >>> >>> > >> 'none', 'BONDING_OPTS': 'mode=4 miimon=100', 'DEVICE': >>>>> >>> >>> > >> 'bond1', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'no'}, 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '9000', >>>>> >>> >>> > >> 'netmask': '255.255.255.0', 'ipv4addrs': ['10.10.10.6/24' >>>>> >>> >>> > >> <http://10.10.10.6/24%27>], 'hwaddr': >>>>> >>> >>> > >> '00:10:18:cd:da:ac', >>>>> >>> >>> > >> 'slaves': >>>>> >>> >>> > >> ['eth2', 'eth3'], 'opts': {'miimon': '100', 'mode': >>>>> >>> >>> > >> '4'}}, >>>>> >>> >>> > >> 'bond2': >>>>> >>> >>> > >> {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', >>>>> >>> >>> > >> 'slaves': >>>>> >>> >>> > >> [], >>>>> >>> >>> > >> 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', >>>>> >>> >>> > >> 'cfg': {}, >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', 'netmask': '', 'slaves': [], 'hwaddr': >>>>> >>> >>> > >> '00:00:00:00:00:00'}}, >>>>> >>> >>> > >> 'software_version': '4.16', 'memSize': '24019', >>>>> >>> >>> > >> 'cpuSpeed': >>>>> >>> >>> > >> '2667.000', >>>>> >>> >>> > >> 'numaNodes': {u'1': {'totalMemory': '12288', 'cpus': [6, >>>>> >>> >>> > >> 7, 8, >>>>> >>> >>> > >> 9, >>>>> >>> >>> > >> 10, 11, >>>>> >>> >>> > >> 18, 19, 20, 21, 22, 23]}, u'0': {'totalMemory': '12278', >>>>> >>> >>> > >> 'cpus': >>>>> >>> >>> > >> [0, >>>>> >>> >>> > >> 1, 2, >>>>> >>> >>> > >> 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, 'version_name': 'Snow >>>>> >>> >>> > >> Man', >>>>> >>> >>> > >> 'vlans': >>>>> >>> >>> > >> {'bond0.10': {'iface': 'bond0', 'addr': '43.25.17.16', >>>>> >>> >>> > >> 'cfg': >>>>> >>> >>> > >> {'DEFROUTE': >>>>> >>> >>> > >> 'yes', 'VLAN': 'yes', 'IPADDR': '43.25.17.16', 'HOTPLUG': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'GATEWAY': >>>>> >>> >>> > >> '43.25.17.1', 'NM_CONTROLLED': 'no', 'NETMASK': >>>>> >>> >>> > >> '255.255.255.0', >>>>> >>> >>> > >> 'BOOTPROTO': 'none', 'DEVICE': 'bond0.10', 'MTU': '1500', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'yes'}, >>>>> >>> >>> > >> 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': >>>>> >>> >>> > >> 10, >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', >>>>> >>> >>> > >> 'netmask': '255.255.255.0', 'ipv4addrs': >>>>> >>> >>> > >> ['43.25.17.16/24'] >>>>> >>> >>> > >> <http://43.25.17.16/24%27%5D>}, 'bond0.36': {'iface': >>>>> >>> >>> > >> 'bond0', >>>>> >>> >>> > >> 'addr': >>>>> >>> >>> > >> '', 'cfg': {'BRIDGE': 'VMNetwork', 'VLAN': 'yes', >>>>> >>> >>> > >> 'HOTPLUG': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'MTU': >>>>> >>> >>> > >> '1500', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond0.36', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'no'}, >>>>> >>> >>> > >> 'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': >>>>> >>> >>> > >> 36, >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '1500', >>>>> >>> >>> > >> 'netmask': '', 'ipv4addrs': []}, 'bond1.100': {'iface': >>>>> >>> >>> > >> 'bond1', >>>>> >>> >>> > >> 'addr': >>>>> >>> >>> > >> '', 'cfg': {'BRIDGE': 'Internal', 'VLAN': 'yes', >>>>> >>> >>> > >> 'HOTPLUG': >>>>> >>> >>> > >> 'no', >>>>> >>> >>> > >> 'MTU': >>>>> >>> >>> > >> '9000', 'NM_CONTROLLED': 'no', 'DEVICE': 'bond1.100', >>>>> >>> >>> > >> 'ONBOOT': >>>>> >>> >>> > >> 'no'}, >>>>> >>> >>> > >> 'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'], 'vlanid': >>>>> >>> >>> > >> 100, >>>>> >>> >>> > >> 'mtu': >>>>> >>> >>> > >> '9000', >>>>> >>> >>> > >> 'netmask': '', 'ipv4addrs': []}}, 'cpuCores': '12', >>>>> >>> >>> > >> 'kvmEnabled': >>>>> >>> >>> > >> 'true', >>>>> >>> >>> > >> 'guestOverhead': '65', 'cpuThreads': '24', >>>>> >>> >>> > >> 'emulatedMachines': >>>>> >>> >>> > >> [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', >>>>> >>> >>> > >> u'rhel6.2.0', >>>>> >>> >>> > >> u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', >>>>> >>> >>> > >> u'rhel5.4.0'], >>>>> >>> >>> > >> 'operatingSystem': {'release': '5.el6.centos.11.1', >>>>> >>> >>> > >> 'version': >>>>> >>> >>> > >> '6', >>>>> >>> >>> > >> 'name': >>>>> >>> >>> > >> 'RHEL'}, 'lastClient': '10.10.10.2'}} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,620::BindingXMLRPC::1132::vds::(wrapper) client >>>>> >>> >>> > >> [10.10.10.2]::call >>>>> >>> >>> > >> getHardwareInfo with () {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,621::BindingXMLRPC::1139::vds::(wrapper) return >>>>> >>> >>> > >> getHardwareInfo >>>>> >>> >>> > >> with {'status': {'message': 'Done', 'code': 0}, 'info': >>>>> >>> >>> > >> {'systemProductName': 'CS24-TY', 'systemSerialNumber': >>>>> >>> >>> > >> '7LWSPN1', >>>>> >>> >>> > >> 'systemFamily': 'Server', 'systemVersion': 'A00', >>>>> >>> >>> > >> 'systemUUID': >>>>> >>> >>> > >> '44454c4c-4c00-1057-8053-b7c04f504e31', >>>>> >>> >>> > >> 'systemManufacturer': >>>>> >>> >>> > >> 'Dell'}} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> 21:41:41,733::BindingXMLRPC::1132::vds::(wrapper) client >>>>> >>> >>> > >> [10.10.10.2]::call >>>>> >>> >>> > >> hostsList with () {} flowID [222e8036] >>>>> >>> >>> > >> Thread-13::ERROR::2014-11-24 >>>>> >>> >>> > >> 21:41:44,753::BindingXMLRPC::1148::vds::(wrapper) vdsm >>>>> >>> >>> > >> exception >>>>> >>> >>> > >> occured >>>>> >>> >>> > >> Traceback (most recent call last): >>>>> >>> >>> > >> File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, >>>>> >>> >>> > >> in >>>>> >>> >>> > >> wrapper >>>>> >>> >>> > >> res = f(*args, **kwargs) >>>>> >>> >>> > >> File "/usr/share/vdsm/gluster/api.py", line 54, in >>>>> >>> >>> > >> wrapper >>>>> >>> >>> > >> rv = func(*args, **kwargs) >>>>> >>> >>> > >> File "/usr/share/vdsm/gluster/api.py", line 251, in >>>>> >>> >>> > >> hostsList >>>>> >>> >>> > >> return {'hosts': self.svdsmProxy.glusterPeerStatus()} >>>>> >>> >>> > >> File "/usr/share/vdsm/supervdsm.py", line 50, in >>>>> >>> >>> > >> __call__ >>>>> >>> >>> > >> return callMethod() >>>>> >>> >>> > >> File "/usr/share/vdsm/supervdsm.py", line 48, in >>>>> >>> >>> > >> <lambda> >>>>> >>> >>> > >> **kwargs) >>>>> >>> >>> > >> File "<string>", line 2, in glusterPeerStatus >>>>> >>> >>> > >> File >>>>> >>> >>> > >> "/usr/lib64/python2.6/multiprocessing/managers.py", line >>>>> >>> >>> > >> 740, >>>>> >>> >>> > >> in >>>>> >>> >>> > >> _callmethod >>>>> >>> >>> > >> raise convert_to_error(kind, result) >>>>> >>> >>> > >> GlusterCmdExecFailedException: Command execution failed >>>>> >>> >>> > >> error: Connection failed. Please check if gluster daemon >>>>> >>> >>> > >> is >>>>> >>> >>> > >> operational. >>>>> >>> >>> > >> return code: 1 >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState) >>>>> >>> >>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from >>>>> >>> >>> > >> state >>>>> >>> >>> > >> init >>>>> >>> >>> > >> -> >>>>> >>> >>> > >> state preparing >>>>> >>> >>> > >> Thread-13::INFO::2014-11-24 >>>>> >>> >>> > >> 21:41:50,950::logUtils::44::dispatcher::(wrapper) Run and >>>>> >>> >>> > >> protect: >>>>> >>> >>> > >> repoStats(options=None) >>>>> >>> >>> > >> Thread-13::INFO::2014-11-24 >>>>> >>> >>> > >> 21:41:50,950::logUtils::47::dispatcher::(wrapper) Run and >>>>> >>> >>> > >> protect: >>>>> >>> >>> > >> repoStats, Return response: {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare) >>>>> >>> >>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState) >>>>> >>> >>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from >>>>> >>> >>> > >> state >>>>> >>> >>> > >> preparing >>>>> >>> >>> > >> -> >>>>> >>> >>> > >> state finished >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) >>>>> >>> >>> > >> Owner.releaseAll requests {} resources {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) >>>>> >>> >>> > >> Owner.cancelAll requests {} >>>>> >>> >>> > >> Thread-13::DEBUG::2014-11-24 >>>>> >>> >>> > >> >>>>> >>> >>> > >> 21:41:50,951::task::993::Storage.TaskManager.Task::(_decref) >>>>> >>> >>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref 0 >>>>> >>> >>> > >> aborting >>>>> >>> >>> > >> False >>>>> >>> >>> > >> ------------------------------- >>>>> >>> >>> > >> >>>>> >>> >>> > >> [root@compute4 ~]# service glusterd status >>>>> >>> >>> > >> glusterd is stopped >>>>> >>> >>> > >> [root@compute4 ~]# chkconfig --list | grep glusterd >>>>> >>> >>> > >> glusterd 0:off 1:off 2:on 3:on 4:on >>>>> >>> >>> > >> 5:on >>>>> >>> >>> > >> 6:off >>>>> >>> >>> > >> [root@compute4 ~]# >>>>> >>> >>> > >> >>>>> >>> >>> > >> Thanks, >>>>> >>> >>> > >> Punit >>>>> >>> >>> > >> >>>>> >>> >>> > >> On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj >>>>> >>> >>> > >> <kmayilsa(a)redhat.com&gt; >>>>> >>> >>> > >> wrote: >>>>> >>> >>> > >> >>>>> >>> >>> > >>> Can you send the corresponding error in vdsm.log from >>>>> >>> >>> > >>> the >>>>> >>> >>> > >>> host? >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> Also check if glusterd service is running. >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> Thanks, >>>>> >>> >>> > >>> Kanagaraj >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> On 11/24/2014 03:39 PM, Punit Dambiwal wrote: >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> Hi, >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> After reboot my Hypervisior host can not activate again >>>>> >>> >>> > >>> in the >>>>> >>> >>> > >>> cluster >>>>> >>> >>> > >>> and failed with the following error :- >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> Gluster command [<UNKNOWN>] failed on server... >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> Engine logs :- >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> 2014-11-24 18:05:28,397 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-64) START, >>>>> >>> >>> > >>> GlusterVolumesListVDSCommand(HostName = Compute4, HostId >>>>> >>> >>> > >>> = >>>>> >>> >>> > >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90 >>>>> >>> >>> > >>> 2014-11-24 18:05:30,609 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-64) FINISH, >>>>> >>> >>> > >>> GlusterVolumesListVDSCommand, >>>>> >>> >>> > >>> return: >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d95203e0}, >>>>> >>> >>> > >>> log id: 5f251c90 >>>>> >>> >>> > >>> 2014-11-24 18:05:33,768 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> >>> >>> > >>> (ajp--127.0.0.1-8702-8) >>>>> >>> >>> > >>> [287d570d] Lock Acquired to object EngineLock >>>>> >>> >>> > >>> [exclusiveLocks= >>>>> >>> >>> > >>> key: >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS >>>>> >>> >>> > >>> , sharedLocks= ] >>>>> >>> >>> > >>> 2014-11-24 18:05:33,795 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] Running >>>>> >>> >>> > >>> command: >>>>> >>> >>> > >>> ActivateVdsCommand internal: false. Entities affected : >>>>> >>> >>> > >>> ID: >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDSAction >>>>> >>> >>> > >>> group >>>>> >>> >>> > >>> MANIPULATE_HOST >>>>> >>> >>> > >>> with role type ADMIN >>>>> >>> >>> > >>> 2014-11-24 18:05:33,796 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] Before >>>>> >>> >>> > >>> acquiring >>>>> >>> >>> > >>> lock in >>>>> >>> >>> > >>> order to prevent monitoring for host Compute5 from >>>>> >>> >>> > >>> data-center >>>>> >>> >>> > >>> SV_WTC >>>>> >>> >>> > >>> 2014-11-24 18:05:33,797 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] Lock >>>>> >>> >>> > >>> acquired, >>>>> >>> >>> > >>> from >>>>> >>> >>> > >>> now a >>>>> >>> >>> > >>> monitoring of host will be skipped for host Compute5 >>>>> >>> >>> > >>> from >>>>> >>> >>> > >>> data-center >>>>> >>> >>> > >>> SV_WTC >>>>> >>> >>> > >>> 2014-11-24 18:05:33,817 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] START, >>>>> >>> >>> > >>> SetVdsStatusVDSCommand(HostName = Compute5, HostId = >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned, >>>>> >>> >>> > >>> nonOperationalReason=NONE, stopSpmFailureLogged=false), >>>>> >>> >>> > >>> log id: >>>>> >>> >>> > >>> 1cbc7311 >>>>> >>> >>> > >>> 2014-11-24 18:05:33,820 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d] FINISH, >>>>> >>> >>> > >>> SetVdsStatusVDSCommand, log id: 1cbc7311 >>>>> >>> >>> > >>> 2014-11-24 18:05:34,086 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) Activate finished. >>>>> >>> >>> > >>> Lock >>>>> >>> >>> > >>> released. >>>>> >>> >>> > >>> Monitoring can run now for host Compute5 from >>>>> >>> >>> > >>> data-center >>>>> >>> >>> > >>> SV_WTC >>>>> >>> >>> > >>> 2014-11-24 18:05:34,088 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) Correlation ID: >>>>> >>> >>> > >>> 287d570d, >>>>> >>> >>> > >>> Job >>>>> >>> >>> > >>> ID: >>>>> >>> >>> > >>> 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call Stack: null, >>>>> >>> >>> > >>> Custom >>>>> >>> >>> > >>> Event ID: >>>>> >>> >>> > >>> -1, Message: Host Compute5 was activated by admin. >>>>> >>> >>> > >>> 2014-11-24 18:05:34,090 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> >>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) Lock freed to object >>>>> >>> >>> > >>> EngineLock >>>>> >>> >>> > >>> [exclusiveLocks= key: >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a >>>>> >>> >>> > >>> value: >>>>> >>> >>> > >>> VDS >>>>> >>> >>> > >>> , sharedLocks= ] >>>>> >>> >>> > >>> 2014-11-24 18:05:35,792 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-55) [3706e836] START, >>>>> >>> >>> > >>> GlusterVolumesListVDSCommand(HostName = Compute4, HostId >>>>> >>> >>> > >>> = >>>>> >>> >>> > >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,064 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) START, >>>>> >>> >>> > >>> GetHardwareInfoVDSCommand(HostName = Compute5, HostId = >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a, >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log >>>>> >>> >>> > >>> id: >>>>> >>> >>> > >>> 6d560cc2 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,074 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) FINISH, >>>>> >>> >>> > >>> GetHardwareInfoVDSCommand, log >>>>> >>> >>> > >>> id: 6d560cc2 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,093 WARN >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.VdsManager] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) Host Compute5 is >>>>> >>> >>> > >>> running >>>>> >>> >>> > >>> with >>>>> >>> >>> > >>> disabled >>>>> >>> >>> > >>> SELinux. >>>>> >>> >>> > >>> 2014-11-24 18:05:37,127 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf] Running >>>>> >>> >>> > >>> command: >>>>> >>> >>> > >>> HandleVdsCpuFlagsOrClusterChangedCommand internal: true. >>>>> >>> >>> > >>> Entities >>>>> >>> >>> > >>> affected >>>>> >>> >>> > >>> : ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS >>>>> >>> >>> > >>> 2014-11-24 18:05:37,147 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf] START, >>>>> >>> >>> > >>> GlusterServersListVDSCommand(HostName = Compute5, HostId >>>>> >>> >>> > >>> = >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,164 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf] FINISH, >>>>> >>> >>> > >>> GlusterServersListVDSCommand, log id: 4faed87 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,189 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] Running >>>>> >>> >>> > >>> command: >>>>> >>> >>> > >>> SetNonOperationalVdsCommand internal: true. Entities >>>>> >>> >>> > >>> affected : >>>>> >>> >>> > >>> ID: >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS >>>>> >>> >>> > >>> 2014-11-24 18:05:37,206 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] START, >>>>> >>> >>> > >>> SetVdsStatusVDSCommand(HostName = Compute5, HostId = >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a, >>>>> >>> >>> > >>> status=NonOperational, >>>>> >>> >>> > >>> nonOperationalReason=GLUSTER_COMMAND_FAILED, >>>>> >>> >>> > >>> stopSpmFailureLogged=false), >>>>> >>> >>> > >>> log id: fed5617 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,209 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] FINISH, >>>>> >>> >>> > >>> SetVdsStatusVDSCommand, log id: fed5617 >>>>> >>> >>> > >>> 2014-11-24 18:05:37,223 ERROR >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] >>>>> >>> >>> > >>> Correlation ID: >>>>> >>> >>> > >>> 4a84c4e5, >>>>> >>> >>> > >>> Job >>>>> >>> >>> > >>> ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b, Call Stack: >>>>> >>> >>> > >>> null, >>>>> >>> >>> > >>> Custom >>>>> >>> >>> > >>> Event >>>>> >>> >>> > >>> ID: -1, Message: Gluster command [<UNKNOWN>] failed on >>>>> >>> >>> > >>> server >>>>> >>> >>> > >>> Compute5. >>>>> >>> >>> > >>> 2014-11-24 18:05:37,243 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5] >>>>> >>> >>> > >>> Correlation ID: >>>>> >>> >>> > >>> null, >>>>> >>> >>> > >>> Call >>>>> >>> >>> > >>> Stack: null, Custom Event ID: -1, Message: Status of >>>>> >>> >>> > >>> host >>>>> >>> >>> > >>> Compute5 >>>>> >>> >>> > >>> was >>>>> >>> >>> > >>> set >>>>> >>> >>> > >>> to NonOperational. >>>>> >>> >>> > >>> 2014-11-24 18:05:37,272 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.bll.HandleVdsVersionCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Running >>>>> >>> >>> > >>> command: >>>>> >>> >>> > >>> HandleVdsVersionCommand internal: true. Entities >>>>> >>> >>> > >>> affected : >>>>> >>> >>> > >>> ID: >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS >>>>> >>> >>> > >>> 2014-11-24 18:05:37,274 INFO >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host >>>>> >>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5 is >>>>> >>> >>> > >>> already in >>>>> >>> >>> > >>> NonOperational status for reason GLUSTER_COMMAND_FAILED. >>>>> >>> >>> > >>> SetNonOperationalVds command is skipped. >>>>> >>> >>> > >>> 2014-11-24 18:05:38,065 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-55) [3706e836] FINISH, >>>>> >>> >>> > >>> GlusterVolumesListVDSCommand, return: >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> {26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4e72a1b1}, >>>>> >>> >>> > >>> log id: 48a0c832 >>>>> >>> >>> > >>> 2014-11-24 18:05:43,243 INFO >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>> >>> >>> > >>> (DefaultQuartzScheduler_Worker-35) START, >>>>> >>> >>> > >>> GlusterVolumesListVDSCommand(HostName = Compute4, HostId >>>>> >>> >>> > >>> = >>>>> >>> >>> > >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc >>>>> >>> >>> > >>> ^C >>>>> >>> >>> > >>> [root@ccr01 ~]# >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> Thanks, >>>>> >>> >>> > >>> Punit >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> _______________________________________________ >>>>> >>> >>> > >>> Users mailing >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >>> >>>>> >>> >>> > >> >>>>> >>> >>> > >> >>>>> >>> >>> > > >>>>> >>> >>> > > >>>>> >>> >>> > >>>>> >>> >> >>>>> >>> >> >>>>> >>> > >>>>> >>> > >>>>> >>> > _______________________________________________ >>>>> >>> > Gluster-users mailing list >>>>> >>> > Gluster-users(a)gluster.org >>>>> >>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users >>>>> >> >>>>> >> >>>>> > >>>> >>>> >>