2013-08-26 20:32:53,573 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) START, GlusterVolumesListVDSCommand(HostName = 172.28.26.104, HostId = 937f9863-fd65-458b-87d1-d01c96d2dfa1), log id: 7b499998 2013-08-26 20:32:53,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7b499998 2013-08-26 20:32:53,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-45) START, GlusterServicesListVDSCommand(HostName = 172.28.26.105, HostId = ddb51c46-a1a7-4b63-8d99-a0472a6f777c), log id: 5f6030ae 2013-08-26 20:32:53,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-41) START, GlusterServicesListVDSCommand(HostName = 172.28.26.104, HostId = 937f9863-fd65-458b-87d1-d01c96d2dfa1), log id: 6eae3b29 2013-08-26 20:32:53,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-49) START, GlusterServicesListVDSCommand(HostName = 172.28.26.102, HostId = dc8ed104-08dd-4235-b00a-d930cb8b646e), log id: 51d8a93b 2013-08-26 20:32:53,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-42) START, GlusterServicesListVDSCommand(HostName = 172.28.26.103, HostId = 563a19bb-a22c-4498-84ca-36e81cdac647), log id: 5564bf9d 2013-08-26 20:32:53,715 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-49) Failed in GlusterServicesListVDS method 2013-08-26 20:32:53,716 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-49) Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to GlusterServicesListVDS, error = Unexpected exception 2013-08-26 20:32:53,717 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-49) Command GlusterServicesListVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GlusterServicesListVDS, error = Unexpected exception 2013-08-26 20:32:53,718 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-49) FINISH, GlusterServicesListVDSCommand, log id: 51d8a93b 2013-08-26 20:32:53,775 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-41) FINISH, GlusterServicesListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@852ca111, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@6bc697f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@b193b40d, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@982ecac2, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@504665d0, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@1b602a8f, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@e679ef4e], log id: 6eae3b29 2013-08-26 20:32:53,781 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-42) FINISH, GlusterServicesListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@698409fa, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@501e00dd, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@95eb1cf6, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@7c8633ab, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@349dceb9, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@ffb79378, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@cad15837], log id: 5564bf9d 2013-08-26 20:32:53,805 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServicesListVDSCommand] (pool-6-thread-45) FINISH, GlusterServicesListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@cc418f28, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@b2db860b, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@f8a8a224, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@df43b8d9, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@975b53e7, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@627518a6, org.ovirt.engine.core.common.businessentities.gluster.GlusterServerService@2d8edd65], log id: 5f6030ae 2013-08-26 20:32:53,807 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (DefaultQuartzScheduler_Worker-15) The thread pool failed to execute list of tasks 2013-08-26 20:32:53,808 ERROR [org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob] (DefaultQuartzScheduler_Worker-15) Error while refreshing service statuses of cluster GlusterCluster!: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GlusterServicesListVDS, error = Unexpected exception (Failed with VDSM error unexpected and code 16) at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil.invokeAll(ThreadPoolUtil.java:179) [utils.jar:] at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:71) [bll.jar:] at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) [:1.7.0_09-icedtea] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Caused by: java.util.concurrent.ExecutionException: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GlusterServicesListVDS, error = Unexpected exception (Failed with VDSM error unexpected and code 16) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.FutureTask.get(FutureTask.java:111) [rt.jar:1.7.0_09-icedtea] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil.invokeAll(ThreadPoolUtil.java:174) [utils.jar:] ... 7 more Caused by: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GlusterServicesListVDS, error = Unexpected exception (Failed with VDSM error unexpected and code 16) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:122) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterJob.runVdsCommand(GlusterJob.java:63) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshServerServices(GlusterServiceSyncJob.java:223) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.access$000(GlusterServiceSyncJob.java:34) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob$1.call(GlusterServiceSyncJob.java:106) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob$1.call(GlusterServiceSyncJob.java:99) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:99) [utils.jar:] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]