[ovirt-users] Re: Gluster problems, cluster performance issues

So, I appear to be at the point where gluster appears to be functional, but engine still won't start. Here's from agent.log on one machine after giving the hosted-engine --vm-start command: MainThread::INFO::2018-05-29 23:52:09,953::hosted_engine::948::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2018-05-29 23:52:10,230::brokerlink::68::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2018-05-29 23:56:16,061::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400) MainThread::INFO::2018-05-29 23:56:16,075::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is unexpectedly down. MainThread::INFO::2018-05-29 23:56:16,076::state_decorators::92::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning <class 'ovirt_hosted_engine_ha.agent.states.EngineStarting'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineMaybeAway'> MainThread::INFO::2018-05-29 23:56:16,359::brokerlink::68::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineMaybeAway) sent? sent MainThread::INFO::2018-05-29 23:56:16,554::ovf_store::151::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 --------------- here's an excerpt from vdsm.log: 2018-05-30 00:00:23,114-0700 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.create succeeded in 0.02 seconds (__init__:573) 2018-05-30 00:00:23,115-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') VM wrapper has started (vm:2764) 2018-05-30 00:00:23,125-0700 INFO (vm/50392390) [vdsm.api] START getVolumeSize(sdUUID='c0acdefb-7d16-48ec-9d76-659b8fe33e2a', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='3de00d8c-d8b0-4ae0-9363-38a504f5d2b2', volUUID='d0fcd8de-6105-4f33-a674-727e3a11e89f', options=None) from=internal, task_id=c6d74ce9-1944-457a-9deb-d0a6929f39ce (api:46) 2018-05-30 00:00:23,130-0700 INFO (vm/50392390) [vdsm.api] FINISH getVolumeSize return={'truesize': '11298571776', 'apparentsize': '10737418240'} from=internal, task_id=c6d74ce9-1944-457a-9deb-d0a6929f39ce (api:52) 2018-05-30 00:00:23,131-0700 INFO (vm/50392390) [vds] prepared volume path: (clientIF:497) 2018-05-30 00:00:23,131-0700 INFO (vm/50392390) [vdsm.api] START prepareImage(sdUUID='c0acdefb-7d16-48ec-9d76-659b8fe33e2a', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='3de00d8c-d8b0-4ae0-9363-38a504f5d2b2', leafUUID='d0fcd8de-6105-4f33-a674-727e3a11e89f', allowIllegal=False) from=internal, task_id=d2797364-270f-4800-92b0-d114441194a3 (api:46) 2018-05-30 00:00:23,155-0700 INFO (vm/50392390) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f (fileSD:622) 2018-05-30 00:00:23,156-0700 INFO (vm/50392390) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a' (fileSD:576) 2018-05-30 00:00:23,156-0700 INFO (vm/50392390) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a mode: None (fileUtils:197) 2018-05-30 00:00:23,157-0700 INFO (vm/50392390) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2 to /var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2 (fileSD:579) 2018-05-30 00:00:23,228-0700 INFO (jsonrpc/0) [root] /usr/libexec/vdsm/hooks/after_get_caps/openstacknet_utils.py: rc=0 err= (hooks:110) 2018-05-30 00:00:23,418-0700 INFO (vm/50392390) [vdsm.api] FINISH prepareImage return={'info': {'path': u'engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f', 'type': 'network', 'hosts': [{'port': '0', 'transport': 'tcp', 'name': '192.168.8.11'}, {'port': '0', 'transport': 'tcp', 'name': ' ovirt1.nwfiber.com'}, {'port': '0', 'transport': 'tcp', 'name': ' ovirt2.nwfiber.com'}, {'port': '0', 'transport': 'tcp', 'name': ' ovirt3.nwfiber.com'}], 'protocol': 'gluster'}, 'path': u'/var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f', 'imgVolumesInfo': [{'domainID': 'c0acdefb-7d16-48ec-9d76-659b8fe33e2a', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f', 'volumeID': u'd0fcd8de-6105-4f33-a674-727e3a11e89f', 'leasePath': u'/rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f.lease', 'imageID': '3de00d8c-d8b0-4ae0-9363-38a504f5d2b2'}]} from=internal, task_id=d2797364-270f-4800-92b0-d114441194a3 (api:52) 2018-05-30 00:00:23,419-0700 INFO (vm/50392390) [vds] prepared volume path: /var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f (clientIF:497) 2018-05-30 00:00:23,420-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') Enabling drive monitoring (drivemonitor:54) 2018-05-30 00:00:23,434-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') drive 'hdc' path: 'file=' -> '*file=' (storagexml:323) 2018-05-30 00:00:23,435-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') drive 'vda' path: 'file=/rhev/data-center/00000000-0000-0000-0000-000000000000/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f' -> u'*file=/var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f' (storagexml:323) 2018-05-30 00:00:23,435-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') Using drive lease: {'domainID': 'c0acdefb-7d16-48ec-9d76-659b8fe33e2a', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f', 'volumeID': u'd0fcd8de-6105-4f33-a674-727e3a11e89f', 'leasePath': u'/rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f.lease', 'imageID': '3de00d8c-d8b0-4ae0-9363-38a504f5d2b2'} (lease:327) 2018-05-30 00:00:23,436-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') lease c0acdefb-7d16-48ec-9d76-659b8fe33e2a:d0fcd8de-6105-4f33-a674-727e3a11e89f path: 'LEASE-PATH:d0fcd8de-6105-4f33-a674-727e3a11e89f:c0acdefb-7d16-48ec-9d76-659b8fe33e2a' -> u'/rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f.lease' (lease:316) 2018-05-30 00:00:23,436-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') lease c0acdefb-7d16-48ec-9d76-659b8fe33e2a:d0fcd8de-6105-4f33-a674-727e3a11e89f offset: 'LEASE-OFFSET:d0fcd8de-6105-4f33-a674-727e3a11e89f:c0acdefb-7d16-48ec-9d76-659b8fe33e2a' -> 0 (lease:316) 2018-05-30 00:00:23,580-0700 INFO (jsonrpc/0) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err=Failed to get Open VSwitch system-id . err = ['ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)'] (hooks:110) 2018-05-30 00:00:23,582-0700 INFO (jsonrpc/0) [api.host] FINISH getCapabilities return={'status': {'message': 'Done', 'code': 0}, 'info': {u'HBAInventory': {u'iSCSI': [{u'InitiatorName': u'iqn.1994-05.com.redhat:1283ce7a3171'}], u'FC': []}, u'packages2': {u'kernel': {u'release': u'862.3.2.el7.x86_64', u'version': u'3.10.0'}, u'glusterfs-rdma': {u'release': u'1.el7', u'version': u'3.12.9'}, u'glusterfs-fuse': {u'release': u'1.el7', u'version': u'3.12.9'}, u'spice-server': {u'release': u'2.el7_5.3', u'version': u'0.14.0'}, u'librbd1': {u'release': u'2.el7', u'version': u'0.94.5'}, u'vdsm': {u'release': u'1.el7.centos', u'version': u'4.20.27.1'}, u'qemu-kvm': {u'release': u'21.el7_5.3.1', u'version': u'2.10.0'}, u'openvswitch': {u'release': u'3.el7', u'version': u'2.9.0'}, u'libvirt': {u'release': u'14.el7_5.5', u'version': u'3.9.0'}, u'ovirt-hosted-engine-ha': {u'release': u'1.el7.centos', u'version': u'2.2.11'}, u'qemu-img': {u'release': u'21.el7_5.3.1', u'version': u'2.10.0'}, u'mom': {u'release': u'1.el7.centos', u'version': u'0.5.12'}, u'glusterfs': {u'release': u'1.el7', u'version': u'3.12.9'}, u'glusterfs-cli': {u'release': u'1.el7', u'version': u'3.12.9'}, u'glusterfs-server': {u'release': u'1.el7', u'version': u'3.12.9'}, u'glusterfs-geo-replication': {u'release': u'1.el7', u'version': u'3.12.9'}}, u'numaNodeDistance': {u'1': [20, 10], u'0': [10, 20]}, u'cpuModel': u'Intel(R) Xeon(R) CPU E5540 @ 2.53GHz', u'nestedVirtualization': False, u'liveMerge': u'true', u'hooks': {u'before_vm_start': {u'50_hostedengine': {u'md5': u'95c810cdcfe4195302a59574a5148289'}, u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}, u'50_vfio_mdev': {u'md5': u'b77eaad7b8f5ca3825dca1b38b0dd446'}}, u'after_network_setup': {u'30_ethtool_options': {u'md5': u'f04c2ca5dce40663e2ed69806eea917c'}}, u'after_vm_destroy': {u'50_vhostmd': {u'md5': u'd70f7ee0453f632e87c09a157ff8ff66'}, u'50_vfio_mdev': {u'md5': u'35364c6e08ecc58410e31af6c73b17f6'}}, u'after_vm_start': {u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'ea0a5a715da8c1badbcda28e8b8fa00e'}}, u'after_device_migrate_destination': {u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'f93c12af7454bbba4c1ef445c2bc9860'}}, u'before_network_setup': {u'50_fcoe': {u'md5': u'28c352339c8beef1e1b05c67d106d062'}}, u'after_get_caps': {u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'0e00c63ab44a952e722209ea31fd7a71'}, u'ovirt_provider_ovn_hook': {u'md5': u'257990644ee6c7824b522e2ab3077ae7'}}, u'after_nic_hotplug': {u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'f93c12af7454bbba4c1ef445c2bc9860'}}, u'before_vm_migrate_destination': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'after_device_create': {u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'f93c12af7454bbba4c1ef445c2bc9860'}}, u'before_vm_dehibernate': {u'50_vhostmd': {u'md5': u'9206bc390bcbf208b06a8e899581be2d'}}, u'before_nic_hotplug': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'0438de5ff9b6bf8d3160804ff71bf827'}, u'ovirt_provider_ovn_hook': {u'md5': u'ada7e4e757fd0241b682b5cc6a545fc8'}}, u'before_device_migrate_destination': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}}, u'before_device_create': {u'50_vmfex': {u'md5': u'49caba1a5faadd8efacef966f79bc30a'}, u'openstacknet_utils.py': {u'md5': u'640ef8b473cbc16b102c5ebf57ea533f'}, u'50_openstacknet': {u'md5': u'0438de5ff9b6bf8d3160804ff71bf827'}, u'ovirt_provider_ovn_hook': {u'md5': u'ada7e4e757fd0241b682b5cc6a545fc8'}}}, u'supportsIPv6': True, u'realtimeKernel': False, u'vmTypes': [u'kvm'], u'liveSnapshot': u'true', u'cpuThreads': u'16', u'kdumpStatus': 0, u'networks': {u'ovirtmgmt': {u'iface': u'ovirtmgmt', u'ipv6autoconf': False, u'addr': u'192.168.8.12', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'mtu': u'1500', u'dhcpv4': True, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u' 192.168.8.12/24'], u'ipv6gateway': u'::', u'gateway': u'192.168.8.1', u'ports': [u'em1']}, u'Public_Fiber': {u'iface': u'Public_Fiber', u'ipv6autoconf': False, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': True, u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u'', u'ports': [u'em1.3']}, u'Gluster': {u'iface': u'em4', u'ipv6autoconf': False, u'addr': u'172.172.1.12', u'dhcpv6': False, u'ipv6addrs': [], u'switch': u'legacy', u'bridged': False, u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': False, u'ipv4addrs': [u' 172.172.1.12/24'], u'interface': u'em4', u'ipv6gateway': u'::', u'gateway': u''}}, u'kernelArgs': u'BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64 root=/dev/mapper/centos_ovirt-root ro crashkernel=auto rd.lvm.lv=centos_ovirt/root rd.lvm.lv=centos_ovirt/swap rhgb quiet LANG=en_US.UTF-8', u'bridges': {u'Public_Fiber': {u'ipv6autoconf': False, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'stp': u'off', u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u'', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.00219b900710', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'25509', u'root_id': u'8000.00219b900710', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'em1.3']}, u'ovirtmgmt': {u'ipv6autoconf': False, u'addr': u'192.168.8.12', u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': True, u'netmask': u'255.255.255.0', u'ipv4defaultroute': True, u'stp': u'off', u'ipv4addrs': [u'192.168.8.12/24'], u'ipv6gateway': u'::', u'gateway': u'192.168.8.1', u'opts': {u'multicast_last_member_count': u'2', u'vlan_protocol': u'0x8100', u'hash_elasticity': u'4', u'multicast_query_response_interval': u'1000', u'group_fwd_mask': u'0x0', u'multicast_snooping': u'1', u'multicast_startup_query_interval': u'3125', u'hello_timer': u'0', u'multicast_querier_interval': u'25500', u'max_age': u'2000', u'hash_max': u'512', u'stp_state': u'0', u'topology_change_detected': u'0', u'priority': u'32768', u'multicast_igmp_version': u'2', u'multicast_membership_interval': u'26000', u'root_path_cost': u'0', u'root_port': u'0', u'multicast_stats_enabled': u'0', u'multicast_startup_query_count': u'2', u'nf_call_iptables': u'0', u'vlan_stats_enabled': u'0', u'hello_time': u'200', u'topology_change': u'0', u'bridge_id': u'8000.00219b900710', u'topology_change_timer': u'0', u'ageing_time': u'30000', u'nf_call_ip6tables': u'0', u'multicast_mld_version': u'1', u'gc_timer': u'13221', u'root_id': u'8000.00219b900710', u'nf_call_arptables': u'0', u'group_addr': u'1:80:c2:0:0:0', u'multicast_last_member_interval': u'100', u'default_pvid': u'1', u'multicast_query_interval': u'12500', u'multicast_query_use_ifaddr': u'0', u'tcn_timer': u'0', u'multicast_router': u'1', u'vlan_filtering': u'0', u'multicast_querier': u'0', u'forward_delay': u'0'}, u'ports': [u'em1']}}, u'uuid': u'4C4C4544-005A-4710-8034-B2C04F4C4B31', u'onlineCpus': u'0,2,4,6,8,10,12,14,1,3,5,7,9,11,13,15', u'nameservers': [u'192.168.8.1', u'8.8.8.8'], u'nics': {u'em4': {u'ipv6autoconf': False, u'addr': u'172.172.1.12', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'255.255.255.0', u'ipv4defaultroute': False, u'ipv4addrs': [u'172.172.1.12/24'], u'hwaddr': u'00:21:9b:90:07:16', u'ipv6gateway': u'::', u'gateway': u''}, u'em1': {u'ipv6autoconf': False, u'addr': u'', u'speed': 1000, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:21:9b:90:07:10', u'ipv6gateway': u'::', u'gateway': u''}, u'em3': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:21:9b:90:07:14', u'ipv6gateway': u'::', u'gateway': u''}, u'em2': {u'ipv6autoconf': True, u'addr': u'', u'speed': 0, u'dhcpv6': False, u'ipv6addrs': [], u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'hwaddr': u'00:21:9b:90:07:12', u'ipv6gateway': u'::', u'gateway': u''}}, u'software_revision': u'1', u'hostdevPassthrough': u'false', u'clusterLevels': [u'3.6', u'4.0', u'4.1', u'4.2'], u'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,popcnt,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,dtherm,ida,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', u'kernelFeatures': {u'RETP': 1, u'IBRS': 0, u'PTI': 1}, u'ISCSIInitiatorName': u'iqn.1994-05.com.redhat:1283ce7a3171', u'netConfigDirty': u'False', u'selinux': {u'mode': u'1'}, u'autoNumaBalancing': 1, u'reservedMem': u'321', u'containers': False, u'bondings': {}, u'software_version': u'4.20', u'supportedENGINEs': [u'3.6', u'4.0', u'4.1', u'4.2'], u'cpuSpeed': u'2527.017', u'numaNodes': {u'1': {u'totalMemory': u'16371', u'cpus': [1, 3, 5, 7, 9, 11, 13, 15]}, u'0': {u'totalMemory': u'16384', u'cpus': [0, 2, 4, 6, 8, 10, 12, 14]}}, u'cpuSockets': u'2', u'vlans': {u'em1.3': {u'iface': u'em1', u'ipv6autoconf': False, u'addr': u'', u'dhcpv6': False, u'ipv6addrs': [], u'vlanid': 3, u'mtu': u'1500', u'dhcpv4': False, u'netmask': u'', u'ipv4defaultroute': False, u'ipv4addrs': [], u'ipv6gateway': u'::', u'gateway': u''}}, u'version_name': u'Snow Man', 'lastClientIface': 'lo', u'cpuCores': u'8', u'hostedEngineDeployed': True, u'hugepages': [2048], u'guestOverhead': u'65', u'additionalFeatures': [u'libgfapi_supported', u'GLUSTER_SNAPSHOT', u'GLUSTER_GEO_REPLICATION', u'GLUSTER_BRICK_MANAGEMENT'], u'kvmEnabled': u'true', u'memSize': u'31996', u'emulatedMachines': [u'pc-i440fx-rhel7.1.0', u'pc-q35-rhel7.3.0', u'rhel6.3.0', u'pc-i440fx-rhel7.5.0', u'pc-i440fx-rhel7.0.0', u'rhel6.1.0', u'pc-i440fx-rhel7.4.0', u'rhel6.6.0', u'pc-q35-rhel7.5.0', u'rhel6.2.0', u'pc', u'pc-i440fx-rhel7.3.0', u'q35', u'pc-i440fx-rhel7.2.0', u'rhel6.4.0', u'pc-q35-rhel7.4.0', u'rhel6.0.0', u'rhel6.5.0'], u'rngSources': [u'hwrng', u'random'], u'operatingSystem': {u'release': u'5.1804.el7.centos.2', u'pretty_name': u'CentOS Linux 7 (Core)', u'version': u'7', u'name': u'RHEL'}}} from=::1,35660 (api:52) 2018-05-30 00:00:23,593-0700 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities succeeded in 3.26 seconds (__init__:573) 2018-05-30 00:00:23,724-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_device_create/50_openstacknet: rc=0 err= (hooks:110) 2018-05-30 00:00:23,919-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_device_create/50_vmfex: rc=0 err= (hooks:110) 2018-05-30 00:00:24,213-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_device_create/openstacknet_utils.py: rc=0 err= (hooks:110) 2018-05-30 00:00:24,392-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_device_create/ovirt_provider_ovn_hook: rc=0 err= (hooks:110) 2018-05-30 00:00:24,401-0700 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,33766 (api:46) 2018-05-30 00:00:24,403-0700 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,33766 (api:52) 2018-05-30 00:00:24,403-0700 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573) 2018-05-30 00:00:24,623-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_vm_start/50_hostedengine: rc=0 err= (hooks:110) 2018-05-30 00:00:24,836-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev: rc=0 err= (hooks:110) 2018-05-30 00:00:25,022-0700 INFO (vm/50392390) [root] /usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd: rc=0 err= (hooks:110) 2018-05-30 00:00:25,023-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') <?xml version="1.0" encoding="utf-8"?><domain type="kvm" xmlns:ns0=" http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> <name>HostedEngine</name> <uuid>50392390-4f89-435c-bf3b-8254b58a4ef7</uuid> <memory>8388608</memory> <currentMemory>8388608</currentMemory> <maxMemory slots="16">33554432</maxMemory> <vcpu current="4">16</vcpu> <sysinfo type="smbios"> <system> <entry name="manufacturer">oVirt</entry> <entry name="product">oVirt Node</entry> <entry name="version">7-5.1804.el7.centos.2</entry> <entry name="serial">4C4C4544-005A-4710-8034-B2C04F4C4B31</entry> <entry name="uuid">50392390-4f89-435c-bf3b-8254b58a4ef7</entry> </system> </sysinfo> <clock adjustment="0" offset="variable"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> </clock> <features> <acpi/> </features> <cpu match="exact"> <model>Nehalem</model> <topology cores="1" sockets="16" threads="1"/> <numa> <cell cpus="0,1,2,3" id="0" memory="8388608"/> </numa> </cpu> <cputune/> <devices> <input bus="usb" type="tablet"/> <channel type="unix"> <target name="ovirt-guest-agent.0" type="virtio"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/50392390-4f89-435c-bf3b-8254b58a4ef7.ovirt-guest-agent.0"/> </channel> <channel type="unix"> <target name="org.qemu.guest_agent.0" type="virtio"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/50392390-4f89-435c-bf3b-8254b58a4ef7.org.qemu.guest_agent.0"/> </channel> <controller index="0" ports="16" type="virtio-serial"> <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/> </controller> <rng model="virtio"> <backend model="random">/dev/urandom</backend> </rng> <graphics autoport="yes" keymap="en-us" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" type="vnc"> <listen network="vdsm-ovirtmgmt" type="network"/> </graphics> <controller type="usb"> <address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"/> </controller> <controller type="ide"> <address bus="0x00" domain="0x0000" function="0x1" slot="0x01" type="pci"/> </controller> <video> <model heads="1" type="cirrus" vram="16384"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x05" type="pci"/> </video> <memballoon model="none"/> <disk device="cdrom" snapshot="no" type="file"> <driver error_policy="report" name="qemu" type="raw"/> <source file="" startupPolicy="optional"/> <target bus="ide" dev="hdc"/> <readonly/> <address bus="1" controller="0" target="0" type="drive" unit="0"/> </disk> <disk device="disk" snapshot="no" type="file"> <target bus="virtio" dev="vda"/> <source file="/var/run/vdsm/storage/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f"/> <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/> <alias name="ua-3de00d8c-d8b0-4ae0-9363-38a504f5d2b2"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci"/> <serial>3de00d8c-d8b0-4ae0-9363-38a504f5d2b2</serial> </disk> <lease> <key>d0fcd8de-6105-4f33-a674-727e3a11e89f</key> <lockspace>c0acdefb-7d16-48ec-9d76-659b8fe33e2a</lockspace> <target offset="0" path="/rhev/data-center/mnt/glusterSD/192.168.8.11: _engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/3de00d8c-d8b0-4ae0-9363-38a504f5d2b2/d0fcd8de-6105-4f33-a674-727e3a11e89f.lease"/> </lease> <interface type="bridge"> <model type="virtio"/> <link state="up"/> <source bridge="ovirtmgmt"/> <alias name="ua-9f93c126-cbb3-4c5b-909e-41a53c2e31fb"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/> <mac address="00:16:3e:2d:bd:b1"/> <filterref filter="vdsm-no-mac-spoofing"/> <bandwidth/> </interface> <channel type="unix"><target name="org.ovirt.hosted-engine-setup.0" type="virtio"/><source mode="bind" path="/var/lib/libvirt/qemu/channels/50392390-4f89-435c-bf3b-8254b58a4ef7.org.ovirt.hosted-engine-setup.0"/></channel></devices> <pm> <suspend-to-disk enabled="no"/> <suspend-to-mem enabled="no"/> </pm> <os> <type arch="x86_64" machine="pc-i440fx-rhel7.3.0">hvm</type> <smbios mode="sysinfo"/> </os> <metadata> <ns0:qos/> <ovirt-vm:vm> <minGuaranteedMemoryMb type="int">8192</minGuaranteedMemoryMb> <clusterVersion>4.2</clusterVersion> <ovirt-vm:custom/> <ovirt-vm:device mac_address="00:16:3e:2d:bd:b1"> <ovirt-vm:custom/> </ovirt-vm:device> <ovirt-vm:device devtype="disk" name="vda"> <ovirt-vm:poolID>00000000-0000-0000-0000-000000000000</ovirt-vm:poolID> <ovirt-vm:volumeID>d0fcd8de-6105-4f33-a674-727e3a11e89f</ovirt-vm:volumeID> <ovirt-vm:shared>exclusive</ovirt-vm:shared> <ovirt-vm:imageID>3de00d8c-d8b0-4ae0-9363-38a504f5d2b2</ovirt-vm:imageID> <ovirt-vm:domainID>c0acdefb-7d16-48ec-9d76-659b8fe33e2a</ovirt-vm:domainID> </ovirt-vm:device> <launchPaused>false</launchPaused> <resumeBehavior>auto_resume</resumeBehavior> </ovirt-vm:vm> </metadata> <on_poweroff>destroy</on_poweroff><on_reboot>destroy</on_reboot><on_crash>destroy</on_crash></domain> (vm:2867) 2018-05-30 00:00:25,252-0700 INFO (monitor/c0acdef) [storage.SANLock] Acquiring host id for domain c0acdefb-7d16-48ec-9d76-659b8fe33e2a (id=2, async=True) (clusterlock:284) 2018-05-30 00:00:26,808-0700 ERROR (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') The vm start process failed (vm:943) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2872, in _run dom.createWithFlags(flags) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirtError: Failed to acquire lock: No space left on device 2018-05-30 00:00:26,809-0700 INFO (vm/50392390) [virt.vm] (vmId='50392390-4f89-435c-bf3b-8254b58a4ef7') Changed state to Down: Failed to acquire lock: No space left on device (code=1) (vm:1683) ------------- The "failed to acquire lock: no space left on device" error has me conserned...Yet there is no device that is full on any servers (espciially the one I'm on): Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_ovirt-root 8.0G 6.6G 1.5G 82% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 4.0K 16G 1% /dev/shm tmpfs 16G 18M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/mapper/gluster-data 177G 59G 119G 33% /gluster/brick2 /dev/mapper/gluster-engine 25G 18G 7.9G 69% /gluster/brick1 /dev/mapper/vg_thin-iso 25G 7.5G 18G 30% /gluster/brick4 /dev/mapper/vg_thin-data--hdd 1.2T 308G 892G 26% /gluster/brick3 /dev/sda1 497M 313M 185M 63% /boot 192.168.8.11:/engine 25G 20G 5.5G 79% /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine tmpfs 3.2G 0 3.2G 0% /run/user/0 192.168.8.11:/data 136G 60G 77G 44% /rhev/data-center/mnt/glusterSD/192.168.8.11:_data 192.168.8.11:/iso 25G 7.5G 18G 30% /rhev/data-center/mnt/glusterSD/192.168.8.11:_iso 172.172.1.11:/data-hdd 1.2T 308G 892G 26% /rhev/data-center/mnt/glusterSD/172.172.1.11:_data-hdd -------- --Jim On Tue, May 29, 2018 at 11:53 PM, Jim Kusznir <jim(a)palousetech.com&gt; wrote: > I've just gotten back to where hsoted-engine --vm-status does > (eventually) return some output other than the ha-agent or storage is down > error. It does currently take up to two minutes, and the last run returned > stale data. > > Gluster-wise, gluster volume heal <volume> info shows the three volumes > with all bricks are all showing 0 entries (and return quickly). The 4th > volume missing one brick (but still has two replicas) does return entries, > and is currently taking a very long time to come back with them. > > Making progress toward getting it online again...Don't know why I get > stale data in hosted-engine --vm-status or how to overcome that... > > --Jim > > > > On Tue, May 29, 2018 at 11:38 PM, Jim Kusznir <jim(a)palousetech.com&gt; > wrote: > >> I believe the gluster data store for the engine is up and working >> correctly. The rest are not mounted, as the engine hasn't managed to start >> correctly yet. I did perform the copy-a-junk-file into the data store, >> then check to ensure its there on another host, then delete that and see >> that it disappeared on the first host; it passed that test. Here's the >> info and status. (I have NOT performed the steps that Krutika and >> Ravishankar suggested yet, as I don't have my data volumes working again >> yet. >> >> [root@ovirt2 images]# gluster volume info >> >> Volume Name: data >> Type: Replicate >> Volume ID: e670c488-ac16-4dd1-8bd3-e43b2e42cc59 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1.nwfiber.com:/gluster/brick2/data >> Brick2: ovirt2.nwfiber.com:/gluster/brick2/data >> Brick3: ovirt3.nwfiber.com:/gluster/brick2/data (arbiter) >> Options Reconfigured: >> diagnostics.count-fop-hits: on >> diagnostics.latency-measurement: on >> changelog.changelog: on >> geo-replication.ignore-pid-check: on >> geo-replication.indexing: on >> server.allow-insecure: on >> performance.readdir-ahead: on >> performance.quick-read: off >> performance.read-ahead: off >> performance.io-cache: off >> performance.stat-prefetch: off >> cluster.eager-lock: enable >> network.remote-dio: enable >> cluster.quorum-type: auto >> cluster.server-quorum-type: server >> storage.owner-uid: 36 >> storage.owner-gid: 36 >> features.shard: on >> features.shard-block-size: 512MB >> performance.low-prio-threads: 32 >> cluster.data-self-heal-algorithm: full >> cluster.locking-scheme: granular >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> network.ping-timeout: 30 >> user.cifs: off >> nfs.disable: on >> performance.strict-o-direct: on >> >> Volume Name: data-hdd >> Type: Replicate >> Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: 172.172.1.11:/gluster/brick3/data-hdd >> Brick2: 172.172.1.12:/gluster/brick3/data-hdd >> Brick3: 172.172.1.13:/gluster/brick3/data-hdd >> Options Reconfigured: >> diagnostics.count-fop-hits: on >> diagnostics.latency-measurement: on >> network.ping-timeout: 30 >> server.allow-insecure: on >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: enable >> performance.low-prio-threads: 32 >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> changelog.changelog: on >> geo-replication.ignore-pid-check: on >> geo-replication.indexing: on >> transport.address-family: inet >> performance.readdir-ahead: on >> >> Volume Name: engine >> Type: Replicate >> Volume ID: 87ad86b9-d88b-457e-ba21-5d3173c612de >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1.nwfiber.com:/gluster/brick1/engine >> Brick2: ovirt2.nwfiber.com:/gluster/brick1/engine >> Brick3: ovirt3.nwfiber.com:/gluster/brick1/engine (arbiter) >> Options Reconfigured: >> changelog.changelog: on >> geo-replication.ignore-pid-check: on >> geo-replication.indexing: on >> performance.readdir-ahead: on >> performance.quick-read: off >> performance.read-ahead: off >> performance.io-cache: off >> performance.stat-prefetch: off >> cluster.eager-lock: enable >> network.remote-dio: off >> cluster.quorum-type: auto >> cluster.server-quorum-type: server >> storage.owner-uid: 36 >> storage.owner-gid: 36 >> features.shard: on >> features.shard-block-size: 512MB >> performance.low-prio-threads: 32 >> cluster.data-self-heal-algorithm: full >> cluster.locking-scheme: granular >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 6 >> network.ping-timeout: 30 >> user.cifs: off >> nfs.disable: on >> performance.strict-o-direct: on >> >> Volume Name: iso >> Type: Replicate >> Volume ID: b1ba15f5-0f0f-4411-89d0-595179f02b92 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1.nwfiber.com:/gluster/brick4/iso >> Brick2: ovirt2.nwfiber.com:/gluster/brick4/iso >> Brick3: ovirt3.nwfiber.com:/gluster/brick4/iso (arbiter) >> Options Reconfigured: >> performance.strict-o-direct: on >> nfs.disable: on >> user.cifs: off >> network.ping-timeout: 30 >> cluster.shd-max-threads: 6 >> cluster.shd-wait-qlength: 10000 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> performance.low-prio-threads: 32 >> features.shard-block-size: 512MB >> features.shard: on >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> network.remote-dio: off >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> performance.readdir-ahead: on >> >> >> [root@ovirt2 images]# gluster volume status >> Status of volume: data >> Gluster process TCP Port RDMA Port Online >> Pid >> >> ------------------------------------------------------------------------------ >> Brick ovirt1.nwfiber.com:/gluster/brick2/da >> ta 49152 0 Y >> 3226 >> Brick ovirt2.nwfiber.com:/gluster/brick2/da >> ta 49152 0 Y >> 2967 >> Brick ovirt3.nwfiber.com:/gluster/brick2/da >> ta 49152 0 Y >> 2554 >> Self-heal Daemon on localhost N/A N/A Y >> 4818 >> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y >> 17853 >> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y >> 4771 >> >> Task Status of Volume data >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Status of volume: data-hdd >> Gluster process TCP Port RDMA Port Online >> Pid >> >> ------------------------------------------------------------------------------ >> Brick 172.172.1.11:/gluster/brick3/data-hdd 49153 0 Y >> 3232 >> Brick 172.172.1.12:/gluster/brick3/data-hdd 49153 0 Y >> 2976 >> Brick 172.172.1.13:/gluster/brick3/data-hdd N/A N/A N >> N/A >> NFS Server on localhost N/A N/A N >> N/A >> Self-heal Daemon on localhost N/A N/A Y >> 4818 >> NFS Server on ovirt3.nwfiber.com N/A N/A N >> N/A >> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y >> 17853 >> NFS Server on ovirt1.nwfiber.com N/A N/A N >> N/A >> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y >> 4771 >> >> Task Status of Volume data-hdd >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Status of volume: engine >> Gluster process TCP Port RDMA Port Online >> Pid >> >> ------------------------------------------------------------------------------ >> Brick ovirt1.nwfiber.com:/gluster/brick1/en >> gine 49154 0 Y >> 3239 >> Brick ovirt2.nwfiber.com:/gluster/brick1/en >> gine 49154 0 Y >> 2982 >> Brick ovirt3.nwfiber.com:/gluster/brick1/en >> gine 49154 0 Y >> 2578 >> Self-heal Daemon on localhost N/A N/A Y >> 4818 >> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y >> 17853 >> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y >> 4771 >> >> Task Status of Volume engine >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Status of volume: iso >> Gluster process TCP Port RDMA Port Online >> Pid >> >> ------------------------------------------------------------------------------ >> Brick ovirt1.nwfiber.com:/gluster/brick4/is >> o 49155 0 Y >> 3247 >> Brick ovirt2.nwfiber.com:/gluster/brick4/is >> o 49155 0 Y >> 2990 >> Brick ovirt3.nwfiber.com:/gluster/brick4/is >> o 49155 0 Y >> 2580 >> Self-heal Daemon on localhost N/A N/A Y >> 4818 >> Self-heal Daemon on ovirt3.nwfiber.com N/A N/A Y >> 17853 >> Self-heal Daemon on ovirt1.nwfiber.com N/A N/A Y >> 4771 >> >> Task Status of Volume iso >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> >> On Tue, May 29, 2018 at 11:30 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >> >>> >>> >>> On Wed, May 30, 2018 at 10:42 AM, Jim Kusznir <jim(a)palousetech.com&gt; >>> wrote: >>> >>>> hosted-engine --deploy failed (would not come up on my existing >>>> gluster storage). However, I realized no changes were written to my >>>> existing storage. So, I went back to trying to get my old engine running. >>>> >>>> hosted-engine --vm-status is now taking a very long time (5+minutes) >>>> to return, and it returns stail information everywhere. I thought perhaps >>>> the lockspace is corrupt, so tried to clean that and metadata, but both are >>>> failing (--cleam-metadata has hung and I can't even ctrl-c out of it). >>>> >>>> How can I reinitialize all the lockspace/metadata safely? There is no >>>> engine or VMs running currently.... >>>> >>> >>> I think the first thing to make sure is that your storage is up and >>> running. So can you mount the gluster volumes and able to access the >>> contents there? >>> Please provide the gluster volume info and gluster volume status of the >>> volumes that you're using. >>> >>> >>> >>>> --Jim >>>> >>>> On Tue, May 29, 2018 at 9:33 PM, Jim Kusznir <jim(a)palousetech.com&gt; >>>> wrote: >>>> >>>>> Well, things went from bad to very, very bad.... >>>>> >>>>> It appears that during one of the 2 minute lockups, the fencing >>>>> agents decided that another node in the cluster was down. As a result, 2 >>>>> of the 3 nodes were simultaneously reset with fencing agent reboot. After >>>>> the nodes came back up, the engine would not start. All running VMs >>>>> (including VMs on the 3rd node that was not rebooted) crashed. >>>>> >>>>> I've now been working for about 3 hours trying to get the engine to >>>>> come up. I don't know why it won't start. hosted-engine --vm-start says >>>>> its starting, but it doesn't start (virsh doesn't show any VMs running). >>>>> I'm currently running --deploy, as I had run out of options for anything >>>>> else I can come up with. I hope this will allow me to re-import all my >>>>> existing VMs and allow me to start them back up after everything comes back >>>>> up. >>>>> >>>>> I do have an unverified geo-rep backup; I don't know if it is a good >>>>> backup (there were several prior messages to this list, but I didn't get >>>>> replies to my questions. It was running in what I believe to be "strange", >>>>> and the data directories are larger than their source). >>>>> >>>>> I'll see if my --deploy works, and if not, I'll be back with another >>>>> message/help request. >>>>> >>>>> When the dust settles and I'm at least minimally functional again, I >>>>> really want to understand why all these technologies designed to offer >>>>> redundancy conspired to reduce uptime and create failures where there >>>>> weren't any otherwise. I thought with hosted engine, 3 ovirt servers and >>>>> glusterfs with minimum replica 2+arb or replica 3 should have offered >>>>> strong resilience against server failure or disk failure, and should have >>>>> prevented / recovered from data corruption. Instead, all of the above >>>>> happened (once I get my cluster back up, I still have to try and recover my >>>>> webserver VM, which won't boot due to XFS corrupt journal issues created >>>>> during the gluster crashes). I think a lot of these issues were rooted >>>>> from the upgrade from 4.1 to 4.2. >>>>> >>>>> --Jim >>>>> >>>>> On Tue, May 29, 2018 at 6:25 PM, Jim Kusznir <jim(a)palousetech.com&gt; >>>>> wrote: >>>>> >>>>>> I also finally found the following in my system log on one server: >>>>>> >>>>>> [10679.524491] INFO: task glusterclogro:14933 blocked for more than >>>>>> 120 seconds. >>>>>> [10679.525826] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [10679.527144] glusterclogro D ffff97209832bf40 0 14933 1 >>>>>> 0x00000080 >>>>>> [10679.527150] Call Trace: >>>>>> [10679.527161] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [10679.527218] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340 >>>>>> [xfs] >>>>>> [10679.527225] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [10679.527254] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs] >>>>>> [10679.527260] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0 >>>>>> [10679.527268] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [10679.527271] [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20 >>>>>> [10679.527275] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [10679.527279] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [10679.527283] INFO: task glusterposixfsy:14941 blocked for more >>>>>> than 120 seconds. >>>>>> [10679.528608] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [10679.529956] glusterposixfsy D ffff972495f84f10 0 14941 1 >>>>>> 0x00000080 >>>>>> [10679.529961] Call Trace: >>>>>> [10679.529966] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [10679.530003] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340 >>>>>> [xfs] >>>>>> [10679.530008] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [10679.530038] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs] >>>>>> [10679.530042] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0 >>>>>> [10679.530046] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [10679.530050] [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20 >>>>>> [10679.530054] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [10679.530058] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [10679.530062] INFO: task glusteriotwr13:15486 blocked for more than >>>>>> 120 seconds. >>>>>> [10679.531805] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [10679.533732] glusteriotwr13 D ffff9720a83f0000 0 15486 1 >>>>>> 0x00000080 >>>>>> [10679.533738] Call Trace: >>>>>> [10679.533747] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [10679.533799] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340 >>>>>> [xfs] >>>>>> [10679.533806] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [10679.533846] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs] >>>>>> [10679.533852] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0 >>>>>> [10679.533858] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [10679.533863] [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20 >>>>>> [10679.533868] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [10679.533873] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [10919.512757] INFO: task glusterclogro:14933 blocked for more than >>>>>> 120 seconds. >>>>>> [10919.514714] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [10919.516663] glusterclogro D ffff97209832bf40 0 14933 1 >>>>>> 0x00000080 >>>>>> [10919.516677] Call Trace: >>>>>> [10919.516690] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [10919.516696] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0 >>>>>> [10919.516703] [<ffffffffb951cc04>] ? blk_finish_plug+0x14/0x40 >>>>>> [10919.516768] [<ffffffffc05e9224>] ? _xfs_buf_ioapply+0x334/0x460 >>>>>> [xfs] >>>>>> [10919.516774] [<ffffffffb991432d>] wait_for_completion+0xfd/0x140 >>>>>> [10919.516782] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [10919.516821] [<ffffffffc05eb0a3>] ? _xfs_buf_read+0x23/0x40 [xfs] >>>>>> [10919.516859] [<ffffffffc05eafa9>] xfs_buf_submit_wait+0xf9/0x1d0 >>>>>> [xfs] >>>>>> [10919.516902] [<ffffffffc061b279>] ? >>>>>> xfs_trans_read_buf_map+0x199/0x400 [xfs] >>>>>> [10919.516940] [<ffffffffc05eb0a3>] _xfs_buf_read+0x23/0x40 [xfs] >>>>>> [10919.516977] [<ffffffffc05eb1b9>] xfs_buf_read_map+0xf9/0x160 >>>>>> [xfs] >>>>>> [10919.517022] [<ffffffffc061b279>] >>>>>> xfs_trans_read_buf_map+0x199/0x400 [xfs] >>>>>> [10919.517057] [<ffffffffc05c8d04>] xfs_da_read_buf+0xd4/0x100 [xfs] >>>>>> [10919.517091] [<ffffffffc05c8d53>] xfs_da3_node_read+0x23/0xd0 >>>>>> [xfs] >>>>>> [10919.517126] [<ffffffffc05c9fee>] >>>>>> xfs_da3_node_lookup_int+0x6e/0x2f0 [xfs] >>>>>> [10919.517160] [<ffffffffc05d5a1d>] xfs_dir2_node_lookup+0x4d/0x170 >>>>>> [xfs] >>>>>> [10919.517194] [<ffffffffc05ccf5d>] xfs_dir_lookup+0x1bd/0x1e0 [xfs] >>>>>> [10919.517233] [<ffffffffc05fd8d9>] xfs_lookup+0x69/0x140 [xfs] >>>>>> [10919.517271] [<ffffffffc05fa018>] xfs_vn_lookup+0x78/0xc0 [xfs] >>>>>> [10919.517278] [<ffffffffb9425cf3>] lookup_real+0x23/0x60 >>>>>> [10919.517283] [<ffffffffb9426702>] __lookup_hash+0x42/0x60 >>>>>> [10919.517288] [<ffffffffb942d519>] SYSC_renameat2+0x3a9/0x5a0 >>>>>> [10919.517296] [<ffffffffb94d3753>] ? >>>>>> selinux_file_free_security+0x23/0x30 >>>>>> [10919.517304] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [10919.517309] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [10919.517313] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [10919.517318] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [10919.517323] [<ffffffffb942e58e>] SyS_renameat2+0xe/0x10 >>>>>> [10919.517328] [<ffffffffb942e5ce>] SyS_rename+0x1e/0x20 >>>>>> [10919.517333] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [10919.517339] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [11159.496095] INFO: task glusteriotwr9:15482 blocked for more than >>>>>> 120 seconds. >>>>>> [11159.497546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [11159.498978] glusteriotwr9 D ffff971fa0fa1fa0 0 15482 1 >>>>>> 0x00000080 >>>>>> [11159.498984] Call Trace: >>>>>> [11159.498995] [<ffffffffb9911f00>] ? bit_wait+0x50/0x50 >>>>>> [11159.498999] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [11159.499003] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0 >>>>>> [11159.499056] [<ffffffffc05dd9b7>] ? >>>>>> xfs_iext_bno_to_ext+0xa7/0x1a0 [xfs] >>>>>> [11159.499082] [<ffffffffc05dd43e>] ? >>>>>> xfs_iext_bno_to_irec+0x8e/0xd0 [xfs] >>>>>> [11159.499090] [<ffffffffb92f7a12>] ? ktime_get_ts64+0x52/0xf0 >>>>>> [11159.499093] [<ffffffffb9911f00>] ? bit_wait+0x50/0x50 >>>>>> [11159.499097] [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130 >>>>>> [11159.499101] [<ffffffffb9913528>] io_schedule+0x18/0x20 >>>>>> [11159.499104] [<ffffffffb9911f11>] bit_wait_io+0x11/0x50 >>>>>> [11159.499107] [<ffffffffb9911ac1>] __wait_on_bit_lock+0x61/0xc0 >>>>>> [11159.499113] [<ffffffffb9393634>] __lock_page+0x74/0x90 >>>>>> [11159.499118] [<ffffffffb92bc210>] ? wake_bit_function+0x40/0x40 >>>>>> [11159.499121] [<ffffffffb9394154>] __find_lock_page+0x54/0x70 >>>>>> [11159.499125] [<ffffffffb9394e85>] >>>>>> grab_cache_page_write_begin+0x55/0xc0 >>>>>> [11159.499130] [<ffffffffb9484b76>] iomap_write_begin+0x66/0x100 >>>>>> [11159.499135] [<ffffffffb9484edf>] iomap_write_actor+0xcf/0x1d0 >>>>>> [11159.499140] [<ffffffffb9484e10>] ? iomap_write_end+0x80/0x80 >>>>>> [11159.499144] [<ffffffffb94854e7>] iomap_apply+0xb7/0x150 >>>>>> [11159.499149] [<ffffffffb9485621>] >>>>>> iomap_file_buffered_write+0xa1/0xe0 >>>>>> [11159.499153] [<ffffffffb9484e10>] ? iomap_write_end+0x80/0x80 >>>>>> [11159.499182] [<ffffffffc05f025d>] >>>>>> xfs_file_buffered_aio_write+0x12d/0x2c0 [xfs] >>>>>> [11159.499213] [<ffffffffc05f057d>] xfs_file_aio_write+0x18d/0x1b0 >>>>>> [xfs] >>>>>> [11159.499217] [<ffffffffb941a533>] do_sync_write+0x93/0xe0 >>>>>> [11159.499222] [<ffffffffb941b010>] vfs_write+0xc0/0x1f0 >>>>>> [11159.499225] [<ffffffffb941c002>] SyS_pwrite64+0x92/0xc0 >>>>>> [11159.499230] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [11159.499234] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [11159.499238] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [11279.488720] INFO: task xfsaild/dm-10:1134 blocked for more than >>>>>> 120 seconds. >>>>>> [11279.490197] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [11279.491665] xfsaild/dm-10 D ffff9720a8660fd0 0 1134 2 >>>>>> 0x00000000 >>>>>> [11279.491671] Call Trace: >>>>>> [11279.491682] [<ffffffffb92a3a2e>] ? >>>>>> try_to_del_timer_sync+0x5e/0x90 >>>>>> [11279.491688] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [11279.491744] [<ffffffffc060de36>] _xfs_log_force+0x1c6/0x2c0 [xfs] >>>>>> [11279.491750] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [11279.491783] [<ffffffffc0619fec>] ? xfsaild+0x16c/0x6f0 [xfs] >>>>>> [11279.491817] [<ffffffffc060df5c>] xfs_log_force+0x2c/0x70 [xfs] >>>>>> [11279.491849] [<ffffffffc0619e80>] ? >>>>>> xfs_trans_ail_cursor_first+0x90/0x90 [xfs] >>>>>> [11279.491880] [<ffffffffc0619fec>] xfsaild+0x16c/0x6f0 [xfs] >>>>>> [11279.491913] [<ffffffffc0619e80>] ? >>>>>> xfs_trans_ail_cursor_first+0x90/0x90 [xfs] >>>>>> [11279.491919] [<ffffffffb92bb161>] kthread+0xd1/0xe0 >>>>>> [11279.491926] [<ffffffffb92bb090>] ? insert_kthread_work+0x40/0x40 >>>>>> [11279.491932] [<ffffffffb9920677>] >>>>>> ret_from_fork_nospec_begin+0x21/0x21 >>>>>> [11279.491936] [<ffffffffb92bb090>] ? insert_kthread_work+0x40/0x40 >>>>>> [11279.491976] INFO: task glusterclogfsyn:14934 blocked for more >>>>>> than 120 seconds. >>>>>> [11279.493466] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [11279.494952] glusterclogfsyn D ffff97209832af70 0 14934 1 >>>>>> 0x00000080 >>>>>> [11279.494957] Call Trace: >>>>>> [11279.494979] [<ffffffffc0309839>] ? >>>>>> __split_and_process_bio+0x2e9/0x520 [dm_mod] >>>>>> [11279.494983] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [11279.494987] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0 >>>>>> [11279.494997] [<ffffffffc0309d98>] ? dm_make_request+0x128/0x1a0 >>>>>> [dm_mod] >>>>>> [11279.495001] [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130 >>>>>> [11279.495005] [<ffffffffb99145ad>] >>>>>> wait_for_completion_io+0xfd/0x140 >>>>>> [11279.495010] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [11279.495016] [<ffffffffb951e574>] blkdev_issue_flush+0xb4/0x110 >>>>>> [11279.495049] [<ffffffffc06064b9>] >>>>>> xfs_blkdev_issue_flush+0x19/0x20 [xfs] >>>>>> [11279.495079] [<ffffffffc05eec40>] xfs_file_fsync+0x1b0/0x1e0 [xfs] >>>>>> [11279.495086] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0 >>>>>> [11279.495090] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [11279.495094] [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20 >>>>>> [11279.495098] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [11279.495102] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [11279.495105] INFO: task glusterposixfsy:14941 blocked for more >>>>>> than 120 seconds. >>>>>> [11279.496606] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [11279.498114] glusterposixfsy D ffff972495f84f10 0 14941 1 >>>>>> 0x00000080 >>>>>> [11279.498118] Call Trace: >>>>>> [11279.498134] [<ffffffffc0309839>] ? >>>>>> __split_and_process_bio+0x2e9/0x520 [dm_mod] >>>>>> [11279.498138] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [11279.498142] [<ffffffffb99118e9>] schedule_timeout+0x239/0x2c0 >>>>>> [11279.498152] [<ffffffffc0309d98>] ? dm_make_request+0x128/0x1a0 >>>>>> [dm_mod] >>>>>> [11279.498156] [<ffffffffb991348d>] io_schedule_timeout+0xad/0x130 >>>>>> [11279.498160] [<ffffffffb99145ad>] >>>>>> wait_for_completion_io+0xfd/0x140 >>>>>> [11279.498165] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [11279.498169] [<ffffffffb951e574>] blkdev_issue_flush+0xb4/0x110 >>>>>> [11279.498202] [<ffffffffc06064b9>] >>>>>> xfs_blkdev_issue_flush+0x19/0x20 [xfs] >>>>>> [11279.498231] [<ffffffffc05eec40>] xfs_file_fsync+0x1b0/0x1e0 [xfs] >>>>>> [11279.498238] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0 >>>>>> [11279.498242] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [11279.498246] [<ffffffffb944f3f3>] SyS_fdatasync+0x13/0x20 >>>>>> [11279.498250] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [11279.498254] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [11279.498257] INFO: task glusteriotwr1:14950 blocked for more than >>>>>> 120 seconds. >>>>>> [11279.499789] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [11279.501343] glusteriotwr1 D ffff97208b6daf70 0 14950 1 >>>>>> 0x00000080 >>>>>> [11279.501348] Call Trace: >>>>>> [11279.501353] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [11279.501390] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340 >>>>>> [xfs] >>>>>> [11279.501396] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [11279.501428] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs] >>>>>> [11279.501432] [<ffffffffb944ef3f>] generic_write_sync+0x4f/0x70 >>>>>> [11279.501461] [<ffffffffc05f0545>] xfs_file_aio_write+0x155/0x1b0 >>>>>> [xfs] >>>>>> [11279.501466] [<ffffffffb941a533>] do_sync_write+0x93/0xe0 >>>>>> [11279.501471] [<ffffffffb941b010>] vfs_write+0xc0/0x1f0 >>>>>> [11279.501475] [<ffffffffb941c002>] SyS_pwrite64+0x92/0xc0 >>>>>> [11279.501479] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [11279.501483] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [11279.501489] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [11279.501493] INFO: task glusteriotwr4:14953 blocked for more than >>>>>> 120 seconds. >>>>>> [11279.503047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [11279.504630] glusteriotwr4 D ffff972499f2bf40 0 14953 1 >>>>>> 0x00000080 >>>>>> [11279.504635] Call Trace: >>>>>> [11279.504640] [<ffffffffb9913f79>] schedule+0x29/0x70 >>>>>> [11279.504676] [<ffffffffc060e388>] _xfs_log_force_lsn+0x2e8/0x340 >>>>>> [xfs] >>>>>> [11279.504681] [<ffffffffb92cf1b0>] ? wake_up_state+0x20/0x20 >>>>>> [11279.504710] [<ffffffffc05eeb97>] xfs_file_fsync+0x107/0x1e0 [xfs] >>>>>> [11279.504714] [<ffffffffb944f0e7>] do_fsync+0x67/0xb0 >>>>>> [11279.504718] [<ffffffffb992076f>] ? >>>>>> system_call_after_swapgs+0xbc/0x160 >>>>>> [11279.504722] [<ffffffffb944f3d0>] SyS_fsync+0x10/0x20 >>>>>> [11279.504725] [<ffffffffb992082f>] system_call_fastpath+0x1c/0x21 >>>>>> [11279.504730] [<ffffffffb992077b>] ? >>>>>> system_call_after_swapgs+0xc8/0x160 >>>>>> [12127.466494] perf: interrupt took too long (8263 > 8150), lowering >>>>>> kernel.perf_event_max_sample_rate to 24000 >>>>>> >>>>>> -------------------- >>>>>> I think this is the cause of the massive ovirt performance issues >>>>>> irrespective of gluster volume. At the time this happened, I was also >>>>>> ssh'ed into the host, and was doing some rpm querry commands. I had just >>>>>> run rpm -qa |grep glusterfs (to verify what version was actually >>>>>> installed), and that command took almost 2 minutes to return! Normally it >>>>>> takes less than 2 seconds. That is all pure local SSD IO, too.... >>>>>> >>>>>> I'm no expert, but its my understanding that anytime a software >>>>>> causes these kinds of issues, its a serious bug in the software, even if >>>>>> its mis-handled exceptions. Is this correct? >>>>>> >>>>>> --Jim >>>>>> >>>>>> On Tue, May 29, 2018 at 3:01 PM, Jim Kusznir <jim(a)palousetech.com&gt; >>>>>> wrote: >>>>>> >>>>>>> I think this is the profile information for one of the volumes that >>>>>>> lives on the SSDs and is fully operational with no down/problem disks: >>>>>>> >>>>>>> [root@ovirt2 yum.repos.d]# gluster volume profile data info >>>>>>> Brick: ovirt2.nwfiber.com:/gluster/brick2/data >>>>>>> ---------------------------------------------- >>>>>>> Cumulative Stats: >>>>>>> Block Size: 256b+ 512b+ >>>>>>> 1024b+ >>>>>>> No. of Reads: 983 2696 >>>>>>> 1059 >>>>>>> No. of Writes: 0 1113 >>>>>>> 302 >>>>>>> >>>>>>> Block Size: 2048b+ 4096b+ >>>>>>> 8192b+ >>>>>>> No. of Reads: 852 88608 >>>>>>> 53526 >>>>>>> No. of Writes: 522 812340 >>>>>>> 76257 >>>>>>> >>>>>>> Block Size: 16384b+ 32768b+ >>>>>>> 65536b+ >>>>>>> No. of Reads: 54351 241901 >>>>>>> 15024 >>>>>>> No. of Writes: 21636 8656 >>>>>>> 8976 >>>>>>> >>>>>>> Block Size: 131072b+ >>>>>>> No. of Reads: 524156 >>>>>>> No. of Writes: 296071 >>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of >>>>>>> calls Fop >>>>>>> --------- ----------- ----------- ----------- >>>>>>> ------------ ---- >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 4189 RELEASE >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 1257 RELEASEDIR >>>>>>> 0.00 46.19 us 12.00 us 187.00 us >>>>>>> 69 FLUSH >>>>>>> 0.00 147.00 us 78.00 us 367.00 us 86 >>>>>>> REMOVEXATTR >>>>>>> 0.00 223.46 us 24.00 us 1166.00 us >>>>>>> 149 READDIR >>>>>>> 0.00 565.34 us 76.00 us 3639.00 us >>>>>>> 88 FTRUNCATE >>>>>>> 0.00 263.28 us 20.00 us 28385.00 us >>>>>>> 228 LK >>>>>>> 0.00 98.84 us 2.00 us 880.00 us >>>>>>> 1198 OPENDIR >>>>>>> 0.00 91.59 us 26.00 us 10371.00 us >>>>>>> 3853 STATFS >>>>>>> 0.00 494.14 us 17.00 us 193439.00 us >>>>>>> 1171 GETXATTR >>>>>>> 0.00 299.42 us 35.00 us 9799.00 us >>>>>>> 2044 READDIRP >>>>>>> 0.00 1965.31 us 110.00 us 382258.00 us >>>>>>> 321 XATTROP >>>>>>> 0.01 113.40 us 24.00 us 61061.00 us >>>>>>> 8134 STAT >>>>>>> 0.01 755.38 us 57.00 us 607603.00 us >>>>>>> 3196 DISCARD >>>>>>> 0.05 2690.09 us 58.00 us 2704761.00 us >>>>>>> 3206 OPEN >>>>>>> 0.10 119978.25 us 97.00 us 9406684.00 us >>>>>>> 154 SETATTR >>>>>>> 0.18 101.73 us 28.00 us 700477.00 us >>>>>>> 313379 FSTAT >>>>>>> 0.23 1059.84 us 25.00 us 2716124.00 us >>>>>>> 38255 LOOKUP >>>>>>> 0.47 1024.11 us 54.00 us 6197164.00 us >>>>>>> 81455 FXATTROP >>>>>>> 1.72 2984.00 us 15.00 us 37098954.00 us >>>>>>> 103020 FINODELK >>>>>>> 5.92 44315.32 us 51.00 us 24731536.00 us >>>>>>> 23957 FSYNC >>>>>>> 13.27 2399.78 us 25.00 us 22089540.00 us >>>>>>> 991005 READ >>>>>>> 37.00 5980.43 us 52.00 us 22099889.00 us >>>>>>> 1108976 WRITE >>>>>>> 41.04 5452.75 us 13.00 us 22102452.00 us >>>>>>> 1349053 INODELK >>>>>>> >>>>>>> Duration: 10026 seconds >>>>>>> Data Read: 80046027759 bytes >>>>>>> Data Written: 44496632320 bytes >>>>>>> >>>>>>> Interval 1 Stats: >>>>>>> Block Size: 256b+ 512b+ >>>>>>> 1024b+ >>>>>>> No. of Reads: 983 2696 >>>>>>> 1059 >>>>>>> No. of Writes: 0 838 >>>>>>> 185 >>>>>>> >>>>>>> Block Size: 2048b+ 4096b+ >>>>>>> 8192b+ >>>>>>> No. of Reads: 852 85856 >>>>>>> 51575 >>>>>>> No. of Writes: 382 705802 >>>>>>> 57812 >>>>>>> >>>>>>> Block Size: 16384b+ 32768b+ >>>>>>> 65536b+ >>>>>>> No. of Reads: 52673 232093 >>>>>>> 14984 >>>>>>> No. of Writes: 13499 4908 >>>>>>> 4242 >>>>>>> >>>>>>> Block Size: 131072b+ >>>>>>> No. of Reads: 460040 >>>>>>> No. of Writes: 6411 >>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of >>>>>>> calls Fop >>>>>>> --------- ----------- ----------- ----------- >>>>>>> ------------ ---- >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 2093 RELEASE >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 1093 RELEASEDIR >>>>>>> 0.00 53.38 us 26.00 us 111.00 us >>>>>>> 16 FLUSH >>>>>>> 0.00 145.14 us 78.00 us 367.00 us 71 >>>>>>> REMOVEXATTR >>>>>>> 0.00 190.96 us 114.00 us 298.00 us >>>>>>> 71 SETATTR >>>>>>> 0.00 213.38 us 24.00 us 1145.00 us >>>>>>> 90 READDIR >>>>>>> 0.00 263.28 us 20.00 us 28385.00 us >>>>>>> 228 LK >>>>>>> 0.00 101.76 us 2.00 us 880.00 us >>>>>>> 1093 OPENDIR >>>>>>> 0.01 93.60 us 27.00 us 10371.00 us >>>>>>> 3090 STATFS >>>>>>> 0.02 537.47 us 17.00 us 193439.00 us >>>>>>> 1038 GETXATTR >>>>>>> 0.03 297.44 us 35.00 us 9799.00 us >>>>>>> 1990 READDIRP >>>>>>> 0.03 2357.28 us 110.00 us 382258.00 us >>>>>>> 253 XATTROP >>>>>>> 0.04 385.93 us 58.00 us 47593.00 us >>>>>>> 2091 OPEN >>>>>>> 0.04 114.86 us 24.00 us 61061.00 us >>>>>>> 7715 STAT >>>>>>> 0.06 444.59 us 57.00 us 333240.00 us >>>>>>> 3053 DISCARD >>>>>>> 0.42 316.24 us 25.00 us 290728.00 us >>>>>>> 29823 LOOKUP >>>>>>> 0.73 257.92 us 54.00 us 344812.00 us >>>>>>> 63296 FXATTROP >>>>>>> 1.37 98.30 us 28.00 us 67621.00 us >>>>>>> 313172 FSTAT >>>>>>> 1.58 2124.69 us 51.00 us 849200.00 us >>>>>>> 16717 FSYNC >>>>>>> 5.73 162.46 us 52.00 us 748492.00 us >>>>>>> 794079 WRITE >>>>>>> 7.19 2065.17 us 16.00 us 37098954.00 us >>>>>>> 78381 FINODELK >>>>>>> 36.44 886.32 us 25.00 us 2216436.00 us >>>>>>> 925421 READ >>>>>>> 46.30 1178.04 us 13.00 us 1700704.00 us >>>>>>> 884635 INODELK >>>>>>> >>>>>>> Duration: 7485 seconds >>>>>>> Data Read: 71250527215 bytes >>>>>>> Data Written: 5119903744 bytes >>>>>>> >>>>>>> Brick: ovirt3.nwfiber.com:/gluster/brick2/data >>>>>>> ---------------------------------------------- >>>>>>> Cumulative Stats: >>>>>>> Block Size: 1b+ >>>>>>> No. of Reads: 0 >>>>>>> No. of Writes: 3264419 >>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of >>>>>>> calls Fop >>>>>>> --------- ----------- ----------- ----------- >>>>>>> ------------ ---- >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 90 FORGET >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 9462 RELEASE >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 4254 RELEASEDIR >>>>>>> 0.00 50.52 us 13.00 us 190.00 us >>>>>>> 71 FLUSH >>>>>>> 0.00 186.97 us 87.00 us 713.00 us 86 >>>>>>> REMOVEXATTR >>>>>>> 0.00 79.32 us 33.00 us 189.00 us >>>>>>> 228 LK >>>>>>> 0.00 220.98 us 129.00 us 513.00 us >>>>>>> 86 SETATTR >>>>>>> 0.01 259.30 us 26.00 us 2632.00 us >>>>>>> 137 READDIR >>>>>>> 0.02 322.76 us 145.00 us 2125.00 us >>>>>>> 321 XATTROP >>>>>>> 0.03 109.55 us 2.00 us 1258.00 us >>>>>>> 1193 OPENDIR >>>>>>> 0.05 70.21 us 21.00 us 431.00 us >>>>>>> 3196 DISCARD >>>>>>> 0.05 169.26 us 21.00 us 2315.00 us >>>>>>> 1545 GETXATTR >>>>>>> 0.12 176.85 us 63.00 us 2844.00 us >>>>>>> 3206 OPEN >>>>>>> 0.61 303.49 us 90.00 us 3085.00 us >>>>>>> 9633 FSTAT >>>>>>> 2.44 305.66 us 28.00 us 3716.00 us >>>>>>> 38230 LOOKUP >>>>>>> 4.52 266.22 us 55.00 us 53424.00 us >>>>>>> 81455 FXATTROP >>>>>>> 6.96 1397.99 us 51.00 us 64822.00 us >>>>>>> 23889 FSYNC >>>>>>> 16.48 84.74 us 25.00 us 6917.00 us >>>>>>> 932592 WRITE >>>>>>> 30.16 106.90 us 13.00 us 3920189.00 us >>>>>>> 1353046 INODELK >>>>>>> 38.55 1794.52 us 14.00 us 16210553.00 us >>>>>>> 103039 FINODELK >>>>>>> >>>>>>> Duration: 66562 seconds >>>>>>> Data Read: 0 bytes >>>>>>> Data Written: 3264419 bytes >>>>>>> >>>>>>> Interval 1 Stats: >>>>>>> Block Size: 1b+ >>>>>>> No. of Reads: 0 >>>>>>> No. of Writes: 794080 >>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of >>>>>>> calls Fop >>>>>>> --------- ----------- ----------- ----------- >>>>>>> ------------ ---- >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 2093 RELEASE >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 1093 RELEASEDIR >>>>>>> 0.00 70.31 us 26.00 us 125.00 us >>>>>>> 16 FLUSH >>>>>>> 0.00 193.10 us 103.00 us 713.00 us 71 >>>>>>> REMOVEXATTR >>>>>>> 0.01 227.32 us 133.00 us 513.00 us >>>>>>> 71 SETATTR >>>>>>> 0.01 79.32 us 33.00 us 189.00 us >>>>>>> 228 LK >>>>>>> 0.01 259.83 us 35.00 us 1138.00 us >>>>>>> 89 READDIR >>>>>>> 0.03 318.26 us 145.00 us 2047.00 us >>>>>>> 253 XATTROP >>>>>>> 0.04 112.67 us 3.00 us 1258.00 us >>>>>>> 1093 OPENDIR >>>>>>> 0.06 167.98 us 23.00 us 1951.00 us >>>>>>> 1014 GETXATTR >>>>>>> 0.08 70.97 us 22.00 us 431.00 us >>>>>>> 3053 DISCARD >>>>>>> 0.13 183.78 us 66.00 us 2844.00 us >>>>>>> 2091 OPEN >>>>>>> 1.01 303.82 us 90.00 us 3085.00 us >>>>>>> 9610 FSTAT >>>>>>> 3.27 316.59 us 30.00 us 3716.00 us >>>>>>> 29820 LOOKUP >>>>>>> 5.83 265.79 us 59.00 us 53424.00 us >>>>>>> 63296 FXATTROP >>>>>>> 7.95 1373.89 us 51.00 us 64822.00 us >>>>>>> 16717 FSYNC >>>>>>> 23.17 851.99 us 14.00 us 16210553.00 us >>>>>>> 78555 FINODELK >>>>>>> 24.04 87.44 us 27.00 us 6917.00 us >>>>>>> 794081 WRITE >>>>>>> 34.36 111.91 us 14.00 us 984871.00 us >>>>>>> 886790 INODELK >>>>>>> >>>>>>> Duration: 7485 seconds >>>>>>> Data Read: 0 bytes >>>>>>> Data Written: 794080 bytes >>>>>>> >>>>>>> >>>>>>> ----------------------- >>>>>>> Here is the data from the volume that is backed by the SHDDs and >>>>>>> has one failed disk: >>>>>>> [root@ovirt2 yum.repos.d]# gluster volume profile data-hdd info >>>>>>> Brick: 172.172.1.12:/gluster/brick3/data-hdd >>>>>>> -------------------------------------------- >>>>>>> Cumulative Stats: >>>>>>> Block Size: 256b+ 512b+ >>>>>>> 1024b+ >>>>>>> No. of Reads: 1702 86 >>>>>>> 16 >>>>>>> No. of Writes: 0 767 >>>>>>> 71 >>>>>>> >>>>>>> Block Size: 2048b+ 4096b+ >>>>>>> 8192b+ >>>>>>> No. of Reads: 19 51841 >>>>>>> 2049 >>>>>>> No. of Writes: 76 60668 >>>>>>> 35727 >>>>>>> >>>>>>> Block Size: 16384b+ 32768b+ >>>>>>> 65536b+ >>>>>>> No. of Reads: 1744 639 >>>>>>> 1088 >>>>>>> No. of Writes: 8524 2410 >>>>>>> 1285 >>>>>>> >>>>>>> Block Size: 131072b+ >>>>>>> No. of Reads: 771999 >>>>>>> No. of Writes: 29584 >>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of >>>>>>> calls Fop >>>>>>> --------- ----------- ----------- ----------- >>>>>>> ------------ ---- >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 2902 RELEASE >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 1517 RELEASEDIR >>>>>>> 0.00 197.00 us 197.00 us 197.00 us >>>>>>> 1 FTRUNCATE >>>>>>> 0.00 70.24 us 16.00 us 758.00 us >>>>>>> 51 FLUSH >>>>>>> 0.00 143.93 us 82.00 us 305.00 us 57 >>>>>>> REMOVEXATTR >>>>>>> 0.00 178.63 us 105.00 us 712.00 us >>>>>>> 60 SETATTR >>>>>>> 0.00 67.30 us 19.00 us 572.00 us >>>>>>> 555 LK >>>>>>> 0.00 322.80 us 23.00 us 4673.00 us >>>>>>> 138 READDIR >>>>>>> 0.00 336.56 us 106.00 us 11994.00 us >>>>>>> 237 XATTROP >>>>>>> 0.00 84.70 us 28.00 us 1071.00 us >>>>>>> 3469 STATFS >>>>>>> 0.01 387.75 us 2.00 us 146017.00 us >>>>>>> 1467 OPENDIR >>>>>>> 0.01 148.59 us 21.00 us 64374.00 us >>>>>>> 4454 STAT >>>>>>> 0.02 783.02 us 16.00 us 93502.00 us >>>>>>> 1902 GETXATTR >>>>>>> 0.03 1516.10 us 17.00 us 210690.00 us >>>>>>> 1364 ENTRYLK >>>>>>> 0.03 2555.47 us 300.00 us 674454.00 us >>>>>>> 1064 READDIRP >>>>>>> 0.07 85.74 us 19.00 us 68340.00 us >>>>>>> 62849 FSTAT >>>>>>> 0.07 1978.12 us 59.00 us 202596.00 us >>>>>>> 2729 OPEN >>>>>>> 0.22 708.57 us 15.00 us 394799.00 us >>>>>>> 25447 LOOKUP >>>>>>> 5.94 2331.74 us 15.00 us 1099530.00 us >>>>>>> 207534 FINODELK >>>>>>> 7.31 8311.75 us 58.00 us 1800216.00 us >>>>>>> 71668 FXATTROP >>>>>>> 12.49 7735.19 us 51.00 us 3595513.00 us >>>>>>> 131642 WRITE >>>>>>> 17.70 957.08 us 16.00 us 13700466.00 us >>>>>>> 1508160 INODELK >>>>>>> 24.55 2546.43 us 26.00 us 5077347.00 us >>>>>>> 786060 READ >>>>>>> 31.56 49699.15 us 47.00 us 3746331.00 us >>>>>>> 51777 FSYNC >>>>>>> >>>>>>> Duration: 10101 seconds >>>>>>> Data Read: 101562897361 bytes >>>>>>> Data Written: 4834450432 bytes >>>>>>> >>>>>>> Interval 0 Stats: >>>>>>> Block Size: 256b+ 512b+ >>>>>>> 1024b+ >>>>>>> No. of Reads: 1702 86 >>>>>>> 16 >>>>>>> No. of Writes: 0 767 >>>>>>> 71 >>>>>>> >>>>>>> Block Size: 2048b+ 4096b+ >>>>>>> 8192b+ >>>>>>> No. of Reads: 19 51841 >>>>>>> 2049 >>>>>>> No. of Writes: 76 60668 >>>>>>> 35727 >>>>>>> >>>>>>> Block Size: 16384b+ 32768b+ >>>>>>> 65536b+ >>>>>>> No. of Reads: 1744 639 >>>>>>> 1088 >>>>>>> No. of Writes: 8524 2410 >>>>>>> 1285 >>>>>>> >>>>>>> Block Size: 131072b+ >>>>>>> No. of Reads: 771999 >>>>>>> No. of Writes: 29584 >>>>>>> %-latency Avg-latency Min-Latency Max-Latency No. of >>>>>>> calls Fop >>>>>>> --------- ----------- ----------- ----------- >>>>>>> ------------ ---- >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 2902 RELEASE >>>>>>> 0.00 0.00 us 0.00 us 0.00 us >>>>>>> 1517 RELEASEDIR >>>>>>> 0.00 197.00 us 197.00 us 197.00 us >>>>>>> 1 FTRUNCATE >>>>>>> 0.00 70.24 us 16.00 us 758.00 us >>>>>>> 51 FLUSH >>>>>>> 0.00 143.93 us 82.00 us 305.00 us 57 >>>>>>> REMOVEXATTR >>>>>>> 0.00 178.63 us 105.00 us 712.00 us >>>>>>> 60 SETATTR >>>>>>> 0.00 67.30 us 19.00 us 572.00 us >>>>>>> 555 LK >>>>>>> 0.00 322.80 us 23.00 us 4673.00 us >>>>>>> 138 READDIR >>>>>>> 0.00 336.56 us 106.00 us 11994.00 us >>>>>>> 237 XATTROP >>>>>>> 0.00 84.70 us 28.00 us 1071.00 us >>>>>>> 3469 STATFS >>>>>>> 0.01 387.75 us 2.00 us 146017.00 us >>>>>>> 1467 OPENDIR >>>>>>> 0.01 148.59 us 21.00 us 64374.00 us >>>>>>> 4454 STAT >>>>>>> 0.02 783.02 us 16.00 us 93502.00 us >>>>>>> 1902 GETXATTR >>>>>>> 0.03 1516.10 us 17.00 us 210690.00 us >>>>>>> 1364 ENTRYLK >>>>>>> 0.03 2555.47 us 300.00 us 674454.00 us >>>>>>> 1064 READDIRP >>>>>>> 0.07 85.73 us 19.00 us 68340.00 us >>>>>>> 62849 FSTAT >>>>>>> 0.07 1978.12 us 59.00 us 202596.00 us >>>>>>> 2729 OPEN >>>>>>> 0.22 708.57 us 15.00 us 394799.00 us >>>>>>> 25447 LOOKUP >>>>>>> 5.94 2334.57 us 15.00 us 1099530.00 us >>>>>>> 207534 FINODELK >>>>>>> 7.31 8311.49 us 58.00 us 1800216.00 us >>>>>>> 71668 FXATTROP >>>>>>> 12.49 7735.32 us 51.00 us 3595513.00 us >>>>>>> 131642 WRITE >>>>>>> 17.71 957.08 us 16.00 us 13700466.00 us >>>>>>> 1508160 INODELK >>>>>>> 24.56 2546.42 us 26.00 us 5077347.00 us >>>>>>> 786060 READ >>>>>>> 31.54 49651.63 us 47.00 us 3746331.00 us >>>>>>> 51777 FSYNC >>>>>>> >>>>>>> Duration: 10101 seconds >>>>>>> Data Read: 101562897361 bytes >>>>>>> Data Written: 4834450432 bytes >>>>>>> >>>>>>> >>>>>>> On Tue, May 29, 2018 at 2:55 PM, Jim Kusznir <jim(a)palousetech.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you for your response. >>>>>>>> >>>>>>>> I have 4 gluster volumes. 3 are >>>>>>>> re_______________________________________________ >>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/B4U3ZMQT5UL... >>>>>>>> >>>>>>>