August 2017 - Devel - oVirt List Archives

Re: [ovirt-devel] vdsm vds.dispatcher
by Gary Pedretty 26 Sep '17

26 Sep '17

--Apple-Mail=_BF0F8909-F772-4228-ADA6-BEDEE7C22B79 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The bug I assumed this was related to is. Bug 1417708 - ovirt-ha-agent should reuse json-rpc connections =20 This is a glusterized 4 host self hosted engine setup. All storage = domains are glusterfs replica 3 volumes on 3 of the 4 hosts that are = part of the cluster. Centos 3.10.0-514.26.2.el7.x86_64 vdsm-4.18.21-1.el7.centos.x86_64 libvirt-2.0.0-10.el7_3.9.x86_64 glusterfs-3.7.20-1.el7.x86_64 qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64 Host and engine are all updated to the latest versions Here are sample of messages log, vdsm,log, supervdsm.log and mom.log messages Aug 28 01:20:11 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:16 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:16 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:19 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:22 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:24 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:37 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:41 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:41 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:44 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:47 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Aug 28 01:20:49 fai-kvm-1 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof vdsm.log Thread-316519::DEBUG::2017-08-28 = 01:21:13,242::bindingxmlrpc::319::vds::(wrapper) client [::1] Thread-316519::DEBUG::2017-08-28 = 01:21:13,242::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::moving from state init -> state preparing Thread-316519::INFO::2017-08-28 = 01:21:13,242::logUtils::49::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone) Thread-316519::INFO::2017-08-28 = 01:21:13,243::logUtils::52::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, = 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000833501', = 'lastCheck': '1.1', 'valid': True}, = u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': False, 'delay': '0.000893509', 'lastCheck': = '1.1', 'valid': True}, u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000776587', 'lastCheck': '1.0', = 'valid': True}, u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, = 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000513211', = 'lastCheck': '1.1', 'valid': True}, = u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000493711', 'lastCheck': = '1.3', 'valid': True}, u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': = True, 'version': 0, 'acquired': True, 'delay': '0.000512272', 'lastCheck': '1.1', 'valid': True}} Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::task::1193::Storage.TaskManager.Task::(prepare) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::finished: = {u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000833501', 'lastCheck': '1.1', 'valid': = True}, u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': = True, 'version': 3, 'acquired': False, 'delay': '0.000893509', = 'lastCheck': '1.1', 'valid': True}, = u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000776587', 'lastCheck': = '1.0', 'valid': True}, u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000513211', 'lastCheck': '1.1', 'valid': True}, = u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000493711', 'lastCheck': '1.3', 'valid': True}, = u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000512272', 'lastCheck': '1.1', 'valid': True}} Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::moving from state preparing -> state finished Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::resourceManager::952::Storage.ResourceManager.Owner::(releas= eAll) Owner.releaseAll requests {} resources {} Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::resourceManager::989::Storage.ResourceManager.Owner::(cancel= All) Owner.cancelAll requests {} Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::task::995::Storage.TaskManager.Task::(_decref) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::ref 0 aborting False Thread-316519::INFO::2017-08-28 = 01:21:13,243::bindingxmlrpc::331::vds::(wrapper) RPC call = storageRepoGetStats finished (code=3D0) in 0.00 seconds Thread-316519::INFO::2017-08-28 = 01:21:13,246::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request = handler for ::1:51992 stopped Thread-68::DEBUG::2017-08-28 = 01:21:14,093::fileSD::159::Storage.StorageDomainManifest::(__init__) = Reading domain in path = /rhev/data-center/mnt/glusterSD/glustermount:data2/2bf8f623-1d82-444e-ae16= -d32e5fe4dc9e Thread-68::DEBUG::2017-08-28 = 01:21:14,093::persistent::194::Storage.PersistentDict::(__init__) = Created a persistent dict with FileMetadataRW backend Thread-68::DEBUG::2017-08-28 = 01:21:14,096::persistent::236::Storage.PersistentDict::(refresh) read = lines (FileMetadataRW)=3D['CLASS=3DData', 'DESCRIPTION=3Ddata2', = 'IOOPTIMEOUTSEC=3D10', 'LEASERETRIES=3D3', 'LEASETIMESEC=3D60', = 'LOCKPOLICY=3D', 'LOCKRENEWALINTERVALSEC=3D5', 'MASTER_VERSION=3D10', = 'POOL_DESCRIPTION=3DDefault', = 'POOL_DOMAINS=3Dcf2f4c75-966b-4f69-90a2-d8e7d21fc052:Active,5e39db25-561f-= 490a-81b6-46a7225f02b6:Active,95d4ed8a-9184-4863-84d5-af2bedd690da:Attache= d,3e04b3ad-c46c-4903-a7b2-b9af681318d9:Attached,bd1f0364-0d3a-44fa-842a-5d= 339caac412:Attached,403df7fb-6cff-46e1-8e04-41213fbc0e6e:Active,2bf8f623-1= d82-444e-ae16-d32e5fe4dc9e:Active,6654cb0c-3e57-42ca-b996-3660a3d32a43:Act= ive,88886a7d-f4cc-45a6-901e-89999aa35d78:Attached,feaebec7-aae3-45b5-9684-= 1feedede7bec:Active', 'POOL_SPM_ID=3D-1', 'POOL_SPM_LVER=3D-1', 'POOL_UUID=3D5990e442-0395-0118-005c-0000000003a1', = 'REMOTE_PATH=3Dglustermount:data2', 'ROLE=3DMaster', = 'SDUUID=3D2bf8f623-1d82-444e-ae16-d32e5fe4dc9e', 'TYPE=3DGLUSTERFS', = 'VERSION=3D3', '_SHA_CKSUM=3D65b61dc4f10d80c67c6133ffaee280919458a06a'] Thread-68::DEBUG::2017-08-28 = 01:21:14,105::fileSD::679::Storage.StorageDomain::(imageGarbageCollector) = Removing remnants of deleted images [] Thread-68::INFO::2017-08-28 = 01:21:14,105::sd::604::Storage.StorageDomain::(_registerResourceNamespaces= ) Resource namespace 2bf8f623-1d82-444e-ae16-d32e5fe4dc9e_imageNS = already registered Thread-68::INFO::2017-08-28 = 01:21:14,105::sd::612::Storage.StorageDomain::(_registerResourceNamespaces= ) Resource namespace 2bf8f623-1d82-444e-ae16-d32e5fe4dc9e_volumeNS = already registered jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,244::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) = Calling 'Host.getStats' in bridge with {} jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,245::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::moving from state init -> = state preparing jsonrpc.Executor/4::INFO::2017-08-28 = 01:21:15,245::logUtils::49::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone) jsonrpc.Executor/4::INFO::2017-08-28 = 01:21:15,245::logUtils::52::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {u'5e39db25-561f-490a-81b6-46a7225f02b6': = {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000833501', 'lastCheck': '3.1', 'valid': True}, = u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': False, 'delay': '0.000893509', 'lastCheck': '3.1', 'valid': True}, = u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000776587', 'lastCheck': = '3.0', 'valid': True}, u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': = True, 'version': 3, 'acquired': True, 'delay': '0.000513211', = 'lastCheck': '1.3', 'valid': True}, u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000493711', 'lastCheck': '1.1', 'valid': True}, = u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000512272', 'lastCheck': '3.1', 'valid': True}} jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::task::1193::Storage.TaskManager.Task::(prepare) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::finished: = {u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000833501', 'lastCheck': = '3.1', 'valid': True}, u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': = 0, 'actual': True, 'version': 3, 'acquired': False, 'delay': '0.000893509', = 'lastCheck': '3.1', 'valid': True}, = u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000776587', 'lastCheck': '3.0', 'valid': True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000513211', 'lastCheck': = '1.3', 'valid': True}, u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': = True, 'version': 3, 'acquired': True, 'delay': '0.000493711', = 'lastCheck': '1.1', 'valid': True}, u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000512272', 'lastCheck': = '3.1', 'valid': True}} jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::moving from state = preparing -> state finished jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::resourceManager::952::Storage.ResourceManager.Owner::(releas= eAll) Owner.releaseAll requests {} resources {} jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::resourceManager::989::Storage.ResourceManager.Owner::(cancel= All) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::task::995::Storage.TaskManager.Task::(_decref) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::ref 0 aborting False Reactor thread::INFO::2017-08-28 = 01:21:15,291::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle= _accept) Accepted connection from ::1:51994 Reactor thread::DEBUG::2017-08-28 = 01:21:15,299::protocoldetector::92::ProtocolDetector.Detector::(__init__) = Using required_size=3D11 Reactor thread::INFO::2017-08-28 = 01:21:15,301::protocoldetector::128::ProtocolDetector.Detector::(handle_re= ad) Detected protocol stomp from ::1:51994 Reactor thread::INFO::2017-08-28 = 01:21:15,301::stompreactor::101::Broker.StompAdapter::(_cmd_connect) = Processing CONNECT request Reactor thread::DEBUG::2017-08-28 = 01:21:15,301::stompreactor::492::protocoldetector.StompDetector::(handle_s= ocket) Stomp detected from ('::1', 51994) JsonRpc (StompReactor)::INFO::2017-08-28 = 01:21:15,313::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) = Subscribe command received jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,356::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) = Return 'Host.getStats' in bridge with {'cpuStatistics': {'1': = {'cpuUser': '2.27', 'nodeIndex': 0, 'cpuSys': '1.80', 'cpuIdle': '95.93'}, '0': {'cpuUser': = '0.67', 'nodeIndex': 0, 'cpuSys': '1.13', 'cpuIdle': '98.20'}, '3': = {'cpuUser': '5.67', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': = '93.73'}, '2': {'cpuUser': '0.93', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '98.54'}, '5': {'cpuUser': '0.87', 'nodeIndex': 0, 'cpuSys': = '1.40', 'cpuIdle': '97.73'}, '4': {'cpuUser': '1.27', 'nodeIndex': 0, = 'cpuSys': '0.53', 'cpuIdle': '98.20'}, '7': {'cpuUser': '0.27', = 'nodeIndex': 0, 'cpuSys': '0.47', 'cpuIdle': '99.26'}, '6': {'cpuUser': '2.07', 'nodeIndex': 0, = 'cpuSys': '0.53', 'cpuIdle': '97.40'}}, 'numaNodeMemFree': {'0': = {'memPercent': 98, 'memFree': '335'}}, 'memShared': 0, 'haScore': 3400, 'thpState': 'always', 'ksmMergeAcrossNodes': True, = 'rxRate': '0.04', 'vmCount': 1, 'memUsed': '41', 'storageDomains': = {u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': False, 'delay': '0.000893509', 'lastCheck': = '3.1', 'valid': True}, u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000776587', 'lastCheck': '3.0', 'valid': True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000513211', = 'lastCheck': '1.3', 'valid': True}, = u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000833501', 'lastCheck': '3.1', 'valid': = True}, u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': = True, 'version': 3, 'acquired': True, 'delay': '0.000493711', = 'lastCheck': '1.1', 'valid': True}, = u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000512272', 'lastCheck': = '3.1', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'enp2s0': {'rxErrors': '0', 'txRate': '0.2', 'rxRate': = '0.1', 'txErrors': '0', 'speed': '1000', 'rxDropped': '18453', 'name': = 'enp2s0', 'tx': '884509425955', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': '134148067121', 'state': 'up'}, 'eno1': = {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '54488653089', 'txDropped': '0', = 'sampleTime': 1503912069.554311, 'rx': '46979728698', 'state': 'up'}, = ';vdsmdummy;': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': = ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '0', 'state': 'down'}, 'enp2s0.4': {'rxErrors': = '0', 'txRate': '0.2', 'rxRate': '0.1', 'txErrors': '0', 'speed': '1000', = 'rxDropped': '0', 'name': 'enp2s0.4', 'tx': '843385870486', 'txDropped': = '0', 'sampleTime': 1503912069.554311, 'rx': '124132253793', 'state': = 'up'}, 'lo': {'rxErrors': '0', 'txRate': '0.3', 'rxRate': '0.3', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': = '444179233658', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': = '444179233658', 'state': 'up'}, 'enp4s0': {'rxErrors': '0', 'txRate': = '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': = '14290', 'name': 'enp4s0', 'tx': '9009494', 'txDropped': '0', 'sampleTime': 1503912069.554311, = 'rx': '187852658', 'state': 'up'}, 'enp4s0.100': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', = 'speed': '1000', 'rxDropped': '0', 'name': 'enp4s0.100', 'tx': = '8798732', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': = '95777939', 'state': 'up'}, 'enp2s0.2': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': = '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': = 'enp2s0.2', 'tx': '1156', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '66329343', 'state': 'up'}, 'enp2s0.3': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp2s0.3', 'tx': '1156', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '5930851', 'state': 'up'}, 'vnet0': = {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', = 'name': 'vnet0', 'tx': '97194738', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '8798642', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', = 'speed': '1000', 'rxDropped': '2', 'name': 'ovirtmgmt', 'tx': = '52893439231', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': = '43528048077', 'state': 'up'}, 'ovirt-3': {'rxErrors': '0', 'txRate': = '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': = '2', 'name': 'ovirt-3', 'tx': '578', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '4621679', 'state': 'up'}, 'ovirt-4': {'rxErrors': '0', 'txRate': '0.2', 'rxRate': = '0.1', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': = 'ovirt-4', 'tx': '843385869908', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': '124132039409', 'state': 'up'}, = 'ovirt-2': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovirt-2', = 'tx': '578', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': = '19462774', 'state': 'up'}, 'dmz': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '4', 'name': 'dmz', 'tx': = '578', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': = '31428052', 'state': 'up'}}, 'txDropped': '0', 'cpuUser': '1.75', 'ksmPages': 1250, = 'elapsedTime': '127838.51', 'cpuLoad': '2.39', 'cpuSys': '0.86', 'diskStats': {'/var/log': {'free': '47378'}, = '/var/log/core': {'free': '47378'}, '/var/run/vdsm/': {'free': '7868'}, = '/tmp': {'free': '47378'}}, 'cpuUserVdsmd': '2.13', 'netConfigDirty': = 'False', 'memCommitted': 2113, 'ksmState': False, 'vmMigrating': 0, = 'ksmCpu': 0, 'memAvailable': 9216, 'txRate': '0.06', 'bootTime': = '1503727419', 'haStats': {'active': True, 'configured': True, 'score': 3400, 'localMaintenance': False, = 'globalMaintenance': False}, 'momStatus': 'active', 'rxDropped': = '32751', 'outgoingVmMigrations': 0, 'swapTotal': 7999, 'swapFree': 7842, 'dateTime': '2017-08-28T09:21:15 GMT', = 'anonHugePages': '2280', 'memFree': 9455, 'cpuIdle': '97.39', = 'vmActive': 1, 'v2vJobs': {}, 'cpuSysVdsmd': '0.87'} jsonrpc.Executor/4::INFO::2017-08-28 = 01:21:15,357::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC = call Host.getStats succeeded in 0.12 seconds jsonrpc.Executor/6::DEBUG::2017-08-28 = 01:21:15,580::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) = Calling 'Host.getHardwareInfo' in bridge with {} jsonrpc.Executor/6::DEBUG::2017-08-28 = 01:21:15,581::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) = Return 'Host.getHardwareInfo' in bridge with {'systemProductName': '', = 'systemUUID': '20DC6410-4F9F-DF11-9A25-7071BC772964', = 'systemSerialNumber': '', 'systemVersion': '', 'systemManufacturer': ''} jsonrpc.Executor/6::INFO::2017-08-28 = 01:21:15,581::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC = call Host.getHardwareInfo succeeded in 0.00 seconds JsonRpc (StompReactor)::ERROR::2017-08-28 = 01:21:15,583::betterAsyncore::113::vds.dispatcher::(recv) SSL error = during reading data: unexpected eof Reactor thread::INFO::2017-08-28 = 01:21:16,325::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle= _accept) Accepted connection from ::1:51996 Reactor thread::DEBUG::2017-08-28 = 01:21:16,332::protocoldetector::92::ProtocolDetector.Detector::(__init__) = Using required_size=3D11 Reactor thread::INFO::2017-08-28 = 01:21:16,332::protocoldetector::128::ProtocolDetector.Detector::(handle_re= ad) Detected protocol xml from ::1:51996 Reactor thread::DEBUG::2017-08-28 = 01:21:16,332::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over = http detected from ('::1', 51996) BindingXMLRPC::INFO::2017-08-28 = 01:21:16,332::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting = request handler for ::1:51996 supervdsm.log MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,517::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev eno1 classid 0:1388 (cwd None) MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,545::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; <rc> = =3D 0 MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,545::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp2s0 classid 0:4 (cwd None) MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,572::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; <rc> = =3D 0 MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,573::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp2s0 classid 0:2 (cwd None) MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,600::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; <rc> = =3D 0 MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,600::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp4s0 classid 0:64 (cwd None) MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,628::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; <rc> = =3D 0 MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,628::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp2s0 classid 0:3 (cwd None) MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,689::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; <rc> = =3D 0 MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,690::vsctl::57::root::(commit) Executing commands: = /usr/bin/ovs-vsctl --oneline --format=3Djson -- list Bridge -- list Port = -- list Interface MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,690::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/bin/ovs-vsctl --oneline --format=3Djson -- list Bridge -- list = Port -- list Interface (cwd None) MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,720::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; <rc> = =3D 0 MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,724::supervdsmServer::99::SuperVdsm.ServerCallback::(wrapper) = return network_caps with {'bridges': {'ovirtmgmt': {'ipv6autoconf': = False, 'addr': '10.9.2.61', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', = 'DNS1': '127.0.0.1', 'IPADDR': '10.9.2.61', 'GATEWAY': '10.9.2.1', = 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', = 'BOOTPROTO': 'none', 'STP': 'off', 'DNS2': '209.193.4.7', 'DEVICE': = 'ovirtmgmt', 'MTU': '1500', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, = 'ipv6addrs': [], 'gateway': '10.9.2.1', 'dhcpv4': False, 'netmask': = '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': = ['10.9.2.61/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eno1'], = 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', = 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', = 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', = 'hello_timer': '104', 'multicast_querier_interval': '25500', 'max_age': = '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': = '0', 'priority': '32768', 'multicast_membership_interval': '26000', = 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.7071bc772964', 'bridge_id': '8000.7071bc772964', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '16704', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'ovirt-4': {'ipv6autoconf': False, 'addr': '10.9.3.61', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'IPADDR': '10.9.3.61', 'MTU': = '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', = 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirt-4', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'gateway': '10.9.3.1', = 'dhcpv4': False, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': = 'off', 'ipv4addrs': ['10.9.3.61/24'], 'mtu': '1500', 'ipv6gateway': = '::', 'ports': ['enp2s0.4'], 'opts': {'multicast_last_member_count': = '2', 'hash_elasticity': '4', 'multicast_query_response_interval': = '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', = 'multicast_startup_query_interval': '3125', 'hello_timer': '102', = 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': = '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': = '32768', 'multicast_membership_interval': '26000', 'root_path_cost': = '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.6805ca4606af', 'bridge_id': '8000.6805ca4606af', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '8307', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'dmz': {'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': = 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': = 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'dmz', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'gateway': '', 'dhcpv4': = False, 'netmask': '', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': [], = 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['vnet0', 'enp4s0.100'], = 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', = 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', = 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', = 'hello_timer': '103', 'multicast_querier_interval': '25500', 'max_age': = '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': = '0', 'priority': '32768', 'multicast_membership_interval': '26000', = 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.001b212f794d', 'bridge_id': '8000.001b212f794d', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '7181', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'ovirt-2': {'ipv6autoconf': False, 'addr': '', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': = 'ovirt-2', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], = 'gateway': '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'stp': = 'off', 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.2'], 'opts': {'multicast_last_member_count': '2', = 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', = 'group_fwd_mask': '0x0', 'multicast_snooping': '1', = 'multicast_startup_query_interval': '3125', 'hello_timer': '2', = 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': = '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': = '32768', 'multicast_membership_interval': '26000', 'root_path_cost': = '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.6805ca4606af', 'bridge_id': '8000.6805ca4606af', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '320', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'ovirt-3': {'ipv6autoconf': False, 'addr': '', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': = 'ovirt-3', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], = 'gateway': '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'stp': = 'off', 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.3'], 'opts': {'multicast_last_member_count': '2', = 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', = 'group_fwd_mask': '0x0', 'multicast_snooping': '1', = 'multicast_startup_query_interval': '3125', 'hello_timer': '1', = 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': = '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': = '32768', 'multicast_membership_interval': '26000', 'root_path_cost': = '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.6805ca4606af', 'bridge_id': '8000.6805ca4606af', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '14656', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}}, 'bondings': {}, 'nameservers': ['127.0.0.1', '209.193.4.7', = '209.112.128.2'], 'nics': {'enp2s0': {'ipv6gateway': '::', = 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': 'no', 'MTU': = '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp2s0', = 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, = 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': = '68:05:ca:46:06:af', 'speed': 1000, 'gateway': ''}, 'eno1': = {'ipv6gateway': '::', 'ipv6autoconf': False, 'addr': '', 'cfg': = {'BRIDGE': 'ovirtmgmt', 'IPV6INIT': 'no', 'MTU': '1500', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'eno1', 'ONBOOT': = 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', = 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '70:71:bc:77:29:64', = 'speed': 1000, 'gateway': ''}, 'enp4s0': {'ipv6gateway': '::', = 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': 'no', 'MTU': = '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp4s0', = 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, = 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': = '00:1b:21:2f:79:4d', 'speed': 1000, 'gateway': ''}}, 'supportsIPv6': = True, 'vlans': {'enp2s0.4': {'iface': 'enp2s0', 'ipv6autoconf': False, = 'addr': '', 'cfg': {'BRIDGE': 'ovirt-4', 'IPV6INIT': 'no', 'VLAN': = 'yes', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', = 'DEVICE': 'enp2s0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'vlanid': 4, = 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, = 'ipv4addrs': [], 'ipv6gateway': '::', 'gateway': ''}, 'enp4s0.100': = {'iface': 'enp4s0', 'ipv6autoconf': False, 'addr': '', 'cfg': {'BRIDGE': = 'dmz', 'IPV6INIT': 'no', 'VLAN': 'yes', 'MTU': '1500', 'NM_CONTROLLED': = 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp4s0.100', 'ONBOOT': 'yes'}, = 'ipv6addrs': [], 'vlanid': 100, 'mtu': '1500', 'dhcpv4': False, = 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'ipv6gateway': '::', = 'gateway': ''}, 'enp2s0.2': {'iface': 'enp2s0', 'ipv6autoconf': False, = 'addr': '', 'cfg': {'BRIDGE': 'ovirt-2', 'IPV6INIT': 'no', 'VLAN': = 'yes', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', = 'DEVICE': 'enp2s0.2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'vlanid': 2, = 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, = 'ipv4addrs': [], 'ipv6gateway': '::', 'gateway': ''}, 'enp2s0.3': = {'iface': 'enp2s0', 'ipv6autoconf': False, 'addr': '', 'cfg': {'BRIDGE': = 'ovirt-3', 'IPV6INIT': 'no', 'VLAN': 'yes', 'MTU': '1500', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp2s0.3', = 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'vlanid': 3, 'mtu': '1500', 'dhcpv4': = False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'ipv6gateway': = '::', 'gateway': ''}}, 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', = 'ipv6autoconf': False, 'addr': '10.9.2.61', 'cfg': {'IPV6INIT': 'no', = 'DEFROUTE': 'yes', 'DNS1': '127.0.0.1', 'IPADDR': '10.9.2.61', 'GATEWAY': = '10.9.2.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': = '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DNS2': = '209.193.4.7', 'DEVICE': 'ovirtmgmt', 'MTU': '1500', 'TYPE': 'Bridge', = 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', = 'gateway': '10.9.2.1', 'dhcpv4': False, 'netmask': '255.255.255.0', = 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': ['10.9.2.61/24'], 'mtu': = '1500', 'ipv6gateway': '::', 'ports': ['eno1']}, 'ovirt-4': {'iface': = 'ovirt-4', 'ipv6autoconf': False, 'addr': '10.9.3.61', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'IPADDR': '10.9.3.61', 'MTU': = '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', = 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirt-4', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': '10.9.3.1', 'dhcpv4': False, 'netmask': = '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': = ['10.9.3.61/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.4']}, 'ovirt-2': {'iface': 'ovirt-2', 'ipv6autoconf': False, = 'addr': '', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'MTU': '1500', = 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off', = 'DEVICE': 'ovirt-2', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': = True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': '', 'dhcpv4': = False, 'netmask': '', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': [], = 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['enp2s0.2']}, 'dmz': = {'iface': 'dmz', 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': = 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': = 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'dmz', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': = 'legacy', 'gateway': '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': = False, 'stp': 'off', 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': = '::', 'ports': ['vnet0', 'enp4s0.100']}, 'ovirt-3': {'iface': 'ovirt-3', = 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': = 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': = 'none', 'STP': 'off', 'DEVICE': 'ovirt-3', 'TYPE': 'Bridge', 'ONBOOT': = 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': = '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'stp': 'off', = 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.3']}}} MainProcess|jsonrpc.Executor/3::DEBUG::2017-08-28 = 01:21:58,027::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper) = call getHardwareInfo with () {} MainProcess|jsonrpc.Executor/3::DEBUG::2017-08-28 = 01:21:58,027::supervdsmServer::99::SuperVdsm.ServerCallback::(wrapper) = return getHardwareInfo with {'systemProductName': '', 'systemUUID': = '20DC6410-4F9F-DF11-9A25-7 ------------------------------------------------------------------------ Gary Pedretty gary(a)ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Green, green as far as the eyes can see yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ > On Aug 27, 2017, at 11:24 PM, Yaniv Kaul <ykaul(a)redhat.com = <mailto:ykaul@redhat.com>> wrote: >=20 > vdsm vds.dispatcher ERROR SSL error during reading data: unexpected = eof --Apple-Mail=_BF0F8909-F772-4228-ADA6-BEDEE7C22B79 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"" style=3D"font-family: LucidaGrande;">The bug = I assumed this was related to is.</div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><span = style=3D"font-family: LucidaGrande;" class=3D"">Bug = 1417708 - ovirt-ha-agent should reuse json-rpc connections =  </span><div class=3D"" style=3D"font-family: LucidaGrande;"><br = class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">This is a glusterized 4 host self hosted engine setup. =  All storage domains are glusterfs replica 3 volumes on 3 of the 4 = hosts that are part of the cluster.</div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><div class=3D"">Centos = 3.10.0-514.26.2.el7.x86_64</div></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: = LucidaGrande;">vdsm-4.18.21-1.el7.centos.x86_64</div><div class=3D"" = style=3D"font-family: = LucidaGrande;">libvirt-2.0.0-10.el7_3.9.x86_64</div><div class=3D"" = style=3D"font-family: = LucidaGrande;">glusterfs-3.7.20-1.el7.x86_64</div><div class=3D"" = style=3D"font-family: = LucidaGrande;">qemu-kvm-tools-ev-2.6.0-28.el7_3.3.1.x86_64</div><div = class=3D"" style=3D"font-family: LucidaGrande;"><br class=3D""></div><div = class=3D"" style=3D"font-family: LucidaGrande;">Host and engine are all = updated to the latest versions</div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">Here are sample of messages log, vdsm,log, supervdsm.log = and mom.log</div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">messages</div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><div class=3D"">Aug 28 01:20:11 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:16 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:16 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:19 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:22 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:24 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:37 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:41 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:41 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:44 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:47 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div><div class=3D"">Aug 28 01:20:49 fai-kvm-1 journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div></div><div class=3D"" style=3D"font-family: LucidaGrande;"><br = class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">vdsm.log</div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><div class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,242::bindingxmlrpc::319::vds::(wrapper) client [::1]</div><div = class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,242::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::moving from = state</div><div class=3D"">init -> state preparing</div><div = class=3D"">Thread-316519::INFO::2017-08-28 = 01:21:13,242::logUtils::49::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone)</div><div = class=3D"">Thread-316519::INFO::2017-08-28 = 01:21:13,243::logUtils::52::dispatcher::(wrapper) Run and protect: = repoStats, Return</div><div class=3D"">response: = {u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000833501', 'lastCheck': = '1.1', 'valid': True}, u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': = 0, 'actual': True, 'version': 3, 'acquired': False, 'delay': = '0.000893509', 'lastCheck': '1.1', 'valid': True}, = u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': 0, 'actual': True, = 'version':</div><div class=3D"">3, 'acquired': True, 'delay': = '0.000776587', 'lastCheck': '1.0', 'valid': True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000513211', 'lastCheck': = '1.1', 'valid': True}, u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000493711', 'lastCheck': '1.3', 'valid':</div><div class=3D"">True}, = u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay':</div><div = class=3D"">'0.000512272', 'lastCheck': '1.1', 'valid': True}}</div><div = class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::task::1193::Storage.TaskManager.Task::(prepare) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::finished: = {u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code':</div><div class=3D"">0,= 'actual': True, 'version': 0,</div><div class=3D"">'acquired': True, = 'delay': '0.000833501', 'lastCheck': '1.1', 'valid': True}, = u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': False, 'delay': '0.000893509', 'lastCheck': = '1.1', 'valid': True}, u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000776587', 'lastCheck': '1.0', 'valid': True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000513211', 'lastCheck': = '1.1', 'valid': True}, = u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e':</div><div class=3D"">{'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000493711', 'lastCheck': '1.3', 'valid': True}, = u'feaebec7-aae3-45b5-9684-1feedede7bec':</div><div class=3D"">{'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000512272', 'lastCheck': '1.1', 'valid': True}}</div><div = class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::moving from = state</div><div class=3D"">preparing -> state finished</div><div = class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::resourceManager::952::Storage.ResourceManager.Owner::(releas= eAll) Owner.releaseAll requests {} resources {}</div><div = class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::resourceManager::989::Storage.ResourceManager.Owner::(cancel= All) Owner.cancelAll requests {}</div><div = class=3D"">Thread-316519::DEBUG::2017-08-28 = 01:21:13,243::task::995::Storage.TaskManager.Task::(_decref) = Task=3D`7c850537-b9e2-4e4b-b346-5fc8ba40f2d1`::ref 0 aborting = False</div><div class=3D"">Thread-316519::INFO::2017-08-28 = 01:21:13,243::bindingxmlrpc::331::vds::(wrapper) RPC call = storageRepoGetStats finished (code=3D0) in 0.00 seconds</div><div = class=3D"">Thread-316519::INFO::2017-08-28 = 01:21:13,246::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request = handler for</div><div class=3D"">::1:51992 stopped</div><div = class=3D"">Thread-68::DEBUG::2017-08-28 = 01:21:14,093::fileSD::159::Storage.StorageDomainManifest::(__init__) = Reading domain in</div><div class=3D"">path = /rhev/data-center/mnt/glusterSD/glustermount:data2/2bf8f623-1d82-444e-ae16= -d32e5fe4dc9e</div><div class=3D"">Thread-68::DEBUG::2017-08-28 = 01:21:14,093::persistent::194::Storage.PersistentDict::(__init__) = Created a persistent</div><div class=3D"">dict with FileMetadataRW = backend</div><div class=3D"">Thread-68::DEBUG::2017-08-28 = 01:21:14,096::persistent::236::Storage.PersistentDict::(refresh) read = lines (FileMetadataRW)=3D['CLASS=3DData', 'DESCRIPTION=3Ddata2', = 'IOOPTIMEOUTSEC=3D10', 'LEASERETRIES=3D3', 'LEASETIMESEC=3D60', = 'LOCKPOLICY=3D', 'LOCKRENEWALINTERVALSEC=3D5', 'MASTER_VERSION=3D10', = 'POOL_DESCRIPTION=3DDefault', = 'POOL_DOMAINS=3Dcf2f4c75-966b-4f69-90a2-d8e7d21fc052:Active,5e39db25-561f-= 490a-81b6-46a7225f02b6:Active,95d4ed8a-9184-4863-84d5-af2bedd690da:Attache= d,3e04b3ad-c46c-4903-a7b2-b9af681318d9:Attached,bd1f0364-0d3a-44fa-842a-5d= 339caac412:Attached,403df7fb-6cff-46e1-8e04-41213fbc0e6e:Active,2bf8f623-1= d82-444e-ae16-d32e5fe4dc9e:Active,6654cb0c-3e57-42ca-b996-3660a3d32a43:Act= ive,88886a7d-f4cc-45a6-901e-89999aa35d78:Attached,feaebec7-aae3-45b5-9684-= 1feedede7bec:Active', 'POOL_SPM_ID=3D-1', 'POOL_SPM_LVER=3D-1',</div><div = class=3D"">'POOL_UUID=3D5990e442-0395-0118-005c-0000000003a1', = 'REMOTE_PATH=3Dglustermount:data2', 'ROLE=3DMaster', = 'SDUUID=3D2bf8f623-1d82-444e-ae16-d32e5fe4dc9e', 'TYPE=3DGLUSTERFS', = 'VERSION=3D3', = '_SHA_CKSUM=3D65b61dc4f10d80c67c6133ffaee280919458a06a']</div><div = class=3D"">Thread-68::DEBUG::2017-08-28 = 01:21:14,105::fileSD::679::Storage.StorageDomain::(imageGarbageCollector) = Removing remnants of deleted images []</div><div = class=3D"">Thread-68::INFO::2017-08-28 = 01:21:14,105::sd::604::Storage.StorageDomain::(_registerResourceNamespaces= ) Resource namespace 2bf8f623-1d82-444e-ae16-d32e5fe4dc9e_imageNS = already registered</div><div class=3D"">Thread-68::INFO::2017-08-28 = 01:21:14,105::sd::612::Storage.StorageDomain::(_registerResourceNamespaces= ) Resource namespace 2bf8f623-1d82-444e-ae16-d32e5fe4dc9e_volumeNS = already registered</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,244::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) = Calling</div><div class=3D"">'Host.getStats' in bridge with {}</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,245::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::moving from state init = -> state preparing</div><div = class=3D"">jsonrpc.Executor/4::INFO::2017-08-28 = 01:21:15,245::logUtils::49::dispatcher::(wrapper) Run and protect: = repoStats(options=3DNone)</div><div = class=3D"">jsonrpc.Executor/4::INFO::2017-08-28 = 01:21:15,245::logUtils::52::dispatcher::(wrapper) Run and protect: = repoStats, Return response: {u'5e39db25-561f-490a-81b6-46a7225f02b6': = {'code': 0, 'actual': True, 'version': 0, 'acquired': True,</div><div = class=3D"">'delay': '0.000833501', 'lastCheck': '3.1', 'valid': True}, = u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': False,</div><div class=3D"">'delay': = '0.000893509', 'lastCheck': '3.1', 'valid': True}, = u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000776587', 'lastCheck': = '3.0', 'valid':</div><div class=3D"">True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000513211', 'lastCheck': = '1.3', 'valid': True},</div><div = class=3D"">u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e':</div><div = class=3D"">{'code': 0, 'actual': True, 'version': 3, 'acquired': True, = 'delay': '0.000493711', 'lastCheck': '1.1', 'valid': True}, = u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, 'actual': True, = 'version': 0, 'acquired':</div><div class=3D"">True, 'delay': = '0.000512272', 'lastCheck': '3.1', 'valid': True}}</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::task::1193::Storage.TaskManager.Task::(prepare) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::finished: = {u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, 'actual': True, = 'version': 0, 'acquired': True, 'delay': '0.000833501', 'lastCheck': = '3.1', 'valid': True}, u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': = 0, 'actual':</div><div class=3D"">True, 'version': 3, 'acquired': False, = 'delay': '0.000893509', 'lastCheck': '3.1', 'valid': True}, = u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True,</div><div class=3D"">'delay': = '0.000776587', 'lastCheck': '3.0', 'valid': True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000513211', 'lastCheck': = '1.3', 'valid':</div><div class=3D"">True}, = u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000493711', 'lastCheck': = '1.1', 'valid': True},</div><div = class=3D"">u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': 0, = 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000512272', = 'lastCheck': '3.1', 'valid': True}}</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::task::597::Storage.TaskManager.Task::(_updateState) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::moving from state = preparing -> state finished</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::resourceManager::952::Storage.ResourceManager.Owner::(releas= eAll) Owner.releaseAll requests {} resources {}</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::resourceManager::989::Storage.ResourceManager.Owner::(cancel= All)</div><div class=3D"">Owner.cancelAll requests {}</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,246::task::995::Storage.TaskManager.Task::(_decref) = Task=3D`5d8c4aff-86cb-447c-a833-e27e5c23c54d`::ref 0 aborting = False</div><div class=3D"">Reactor thread::INFO::2017-08-28 = 01:21:15,291::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle= _accept)</div><div class=3D"">Accepted connection from = ::1:51994</div><div class=3D"">Reactor thread::DEBUG::2017-08-28 = 01:21:15,299::protocoldetector::92::ProtocolDetector.Detector::(__init__) = Using required_size=3D11</div><div class=3D"">Reactor = thread::INFO::2017-08-28 = 01:21:15,301::protocoldetector::128::ProtocolDetector.Detector::(handle_re= ad) Detected protocol stomp from ::1:51994</div><div class=3D"">Reactor = thread::INFO::2017-08-28 = 01:21:15,301::stompreactor::101::Broker.StompAdapter::(_cmd_connect) = Processing CONNECT request</div><div class=3D"">Reactor = thread::DEBUG::2017-08-28 = 01:21:15,301::stompreactor::492::protocoldetector.StompDetector::(handle_s= ocket) Stomp detected from ('::1', 51994)</div><div class=3D"">JsonRpc = (StompReactor)::INFO::2017-08-28 = 01:21:15,313::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) = Subscribe command received</div><div = class=3D"">jsonrpc.Executor/4::DEBUG::2017-08-28 = 01:21:15,356::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) = Return 'Host.getStats' in bridge with {'cpuStatistics': {'1': = {'cpuUser': '2.27',</div><div class=3D"">'nodeIndex': 0, 'cpuSys': = '1.80', 'cpuIdle': '95.93'}, '0': {'cpuUser': '0.67', 'nodeIndex': 0, = 'cpuSys': '1.13', 'cpuIdle': '98.20'}, '3': {'cpuUser': '5.67', = 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '93.73'},</div><div = class=3D"">'2': {'cpuUser': '0.93', 'nodeIndex': 0, 'cpuSys': = '0.53',</div><div class=3D"">'cpuIdle': '98.54'}, '5': {'cpuUser': = '0.87', 'nodeIndex': 0, 'cpuSys': '1.40', 'cpuIdle': '97.73'}, '4': = {'cpuUser': '1.27', 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': = '98.20'}, '7': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys':</div><div = class=3D"">'0.47', 'cpuIdle': '99.26'}, '6': {'cpuUser': '2.07', = 'nodeIndex': 0, 'cpuSys': '0.53', 'cpuIdle': '97.40'}}, = 'numaNodeMemFree': {'0': {'memPercent': 98, 'memFree': '335'}}, = 'memShared': 0,</div><div class=3D"">'haScore': 3400, 'thpState': = 'always', 'ksmMergeAcrossNodes': True, 'rxRate': '0.04', 'vmCount': 1, = 'memUsed': '41', 'storageDomains': = {u'403df7fb-6cff-46e1-8e04-41213fbc0e6e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': False, 'delay': '0.000893509', 'lastCheck': = '3.1', 'valid': True}, u'cf2f4c75-966b-4f69-90a2-d8e7d21fc052': {'code': = 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000776587', 'lastCheck': '3.0', 'valid': True}, = u'6654cb0c-3e57-42ca-b996-3660a3d32a43': {'code': 0,</div><div = class=3D"">'actual': True, 'version': 3, 'acquired': True, 'delay': = '0.000513211', 'lastCheck': '1.3', 'valid': True}, = u'5e39db25-561f-490a-81b6-46a7225f02b6': {'code': 0, 'actual': True, = 'version': 0,</div><div class=3D"">'acquired': True, 'delay': = '0.000833501', 'lastCheck': '3.1', 'valid': True}, = u'2bf8f623-1d82-444e-ae16-d32e5fe4dc9e': {'code': 0, 'actual': True, = 'version': 3, 'acquired': True, 'delay': '0.000493711', 'lastCheck': = '1.1', 'valid': True}, u'feaebec7-aae3-45b5-9684-1feedede7bec': {'code': = 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': = '0.000512272', 'lastCheck': '3.1', 'valid': True}}, = 'incomingVmMigrations': 0,</div><div class=3D"">'network': {'enp2s0': = {'rxErrors': '0', 'txRate': '0.2', 'rxRate': '0.1', 'txErrors': '0', = 'speed': '1000', 'rxDropped': '18453', 'name': 'enp2s0', 'tx': = '884509425955', 'txDropped': '0', 'sampleTime':</div><div = class=3D"">1503912069.554311, 'rx': '134148067121', 'state': 'up'}, = 'eno1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', = 'txErrors':</div><div class=3D"">'0', 'speed': '1000',</div><div = class=3D"">'rxDropped': '0', 'name': 'eno1', 'tx': '54488653089', = 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': '46979728698', = 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txRate': '0.0', = 'rxRate': '0.0',</div><div class=3D"">'txErrors': '0', 'speed': '1000', = 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', = 'sampleTime': 1503912069.554311, 'rx': '0', 'state': 'down'}, = 'enp2s0.4': {'rxErrors': '0', 'txRate': '0.2', 'rxRate': '0.1', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp2s0.4', = 'tx': '843385870486', 'txDropped': '0', 'sampleTime': 1503912069.554311, = 'rx': '124132253793', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txRate': = '0.3', 'rxRate': '0.3', 'txErrors': '0', 'speed': '1000', 'rxDropped': = '0', 'name': 'lo', 'tx': '444179233658', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '444179233658', 'state': 'up'}, 'enp4s0': = {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', = 'speed': '1000', 'rxDropped': '14290', 'name': 'enp4s0',</div><div = class=3D"">'tx': '9009494', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '187852658', 'state': 'up'}, = 'enp4s0.100':</div><div class=3D"">{'rxErrors': '0', 'txRate': '0.0', = 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', = 'name': 'enp4s0.100', 'tx': '8798732', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '95777939',</div><div class=3D"">'state': = 'up'}, 'enp2s0.2': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'enp2s0.2', = 'tx': '1156', 'txDropped': '0', 'sampleTime': 1503912069.554311, 'rx': = '66329343', 'state': 'up'},</div><div class=3D"">'enp2s0.3': = {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', = 'speed': '1000', 'rxDropped': '0',</div><div class=3D"">'name': = 'enp2s0.3', 'tx': '1156', 'txDropped': '0', 'sampleTime': = 1503912069.554311, 'rx': '5930851', 'state': 'up'}, 'vnet0': = {'rxErrors': '0', 'txRate': '0.0',</div><div class=3D"">'rxRate': '0.0', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'vnet0', = 'tx': '97194738', 'txDropped': '0', 'sampleTime': 1503912069.554311, = 'rx': '8798642', 'state': 'up'}, 'ovirtmgmt':</div><div = class=3D"">{'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', = 'txErrors': '0', 'speed': '1000', 'rxDropped': '2', 'name': 'ovirtmgmt', = 'tx': '52893439231', 'txDropped': '0', 'sampleTime': 1503912069.554311, = 'rx': '43528048077', 'state': 'up'}, 'ovirt-3': {'rxErrors': '0', = 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', = 'rxDropped': '2', 'name': 'ovirt-3', 'tx': '578', 'txDropped': '0', = 'sampleTime': 1503912069.554311, 'rx': '4621679',</div><div = class=3D"">'state': 'up'}, 'ovirt-4': {'rxErrors': '0', 'txRate': '0.2', = 'rxRate': '0.1', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', = 'name': 'ovirt-4', 'tx': '843385869908', 'txDropped': '0',</div><div = class=3D"">'sampleTime': 1503912069.554311, 'rx': '124132039409', = 'state': 'up'}, 'ovirt-2': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': = '0.0',</div><div class=3D"">'txErrors': '0', 'speed': '1000', = 'rxDropped': '0', 'name': 'ovirt-2', 'tx': '578', 'txDropped': '0', = 'sampleTime': 1503912069.554311, 'rx': '19462774', 'state':</div><div = class=3D"">'up'}, 'dmz': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': = '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '4', 'name': = 'dmz', 'tx': '578', 'txDropped': '0', 'sampleTime': 1503912069.554311, = 'rx': '31428052',</div><div class=3D"">'state': 'up'}}, 'txDropped': = '0', 'cpuUser': '1.75', 'ksmPages': 1250, 'elapsedTime': '127838.51', = 'cpuLoad':</div><div class=3D"">'2.39', 'cpuSys': '0.86', 'diskStats': = {'/var/log': {'free': '47378'}, '/var/log/core': {'free': '47378'}, = '/var/run/vdsm/': {'free': '7868'}, '/tmp': {'free': '47378'}}, = 'cpuUserVdsmd': '2.13', 'netConfigDirty': 'False', 'memCommitted': 2113, = 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 9216, = 'txRate': '0.06', 'bootTime': '1503727419', 'haStats': = {'active':</div><div class=3D"">True, 'configured': True, 'score': 3400, = 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': = 'active', 'rxDropped': '32751', 'outgoingVmMigrations': 0, 'swapTotal': = 7999,</div><div class=3D"">'swapFree': 7842, 'dateTime': = '2017-08-28T09:21:15 GMT', 'anonHugePages': '2280', 'memFree': 9455, = 'cpuIdle': '97.39', 'vmActive': 1, 'v2vJobs': {}, 'cpuSysVdsmd': = '0.87'}</div><div class=3D"">jsonrpc.Executor/4::INFO::2017-08-28 = 01:21:15,357::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC = call Host.getStats succeeded in 0.12 seconds</div><div = class=3D"">jsonrpc.Executor/6::DEBUG::2017-08-28 = 01:21:15,580::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) = Calling</div><div class=3D"">'Host.getHardwareInfo' in bridge with = {}</div><div class=3D"">jsonrpc.Executor/6::DEBUG::2017-08-28 = 01:21:15,581::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) = Return 'Host.getHardwareInfo' in bridge with {'systemProductName': '', = 'systemUUID': '20DC6410-4F9F-DF11-9A25-7071BC772964', = 'systemSerialNumber': '', 'systemVersion': '', 'systemManufacturer': = ''}</div><div class=3D"">jsonrpc.Executor/6::INFO::2017-08-28 = 01:21:15,581::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC = call Host.getHardwareInfo succeeded in 0.00 seconds</div><div = class=3D"">JsonRpc (StompReactor)::ERROR::2017-08-28 = 01:21:15,583::betterAsyncore::113::vds.dispatcher::(recv) SSL error = during</div><div class=3D"">reading data: unexpected eof</div><div = class=3D"">Reactor thread::INFO::2017-08-28 = 01:21:16,325::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle= _accept)</div><div class=3D"">Accepted connection from = ::1:51996</div><div class=3D"">Reactor thread::DEBUG::2017-08-28 = 01:21:16,332::protocoldetector::92::ProtocolDetector.Detector::(__init__) = Using required_size=3D11</div><div class=3D"">Reactor = thread::INFO::2017-08-28 = 01:21:16,332::protocoldetector::128::ProtocolDetector.Detector::(handle_re= ad) Detected protocol xml from ::1:51996</div><div class=3D"">Reactor = thread::DEBUG::2017-08-28 = 01:21:16,332::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over = http detected from ('::1', 51996)</div><div = class=3D"">BindingXMLRPC::INFO::2017-08-28 = 01:21:16,332::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting = request handler for ::1:51996</div></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;">supervdsm.log</div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,517::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev eno1 classid 0:1388 (cwd None)</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,545::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; = <rc> =3D 0</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,545::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp2s0 classid 0:4 (cwd None)</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,572::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; = <rc> =3D 0</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,573::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp2s0 classid 0:2 (cwd None)</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,600::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; = <rc> =3D 0</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,600::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp4s0 classid 0:64 (cwd None)</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,628::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; = <rc> =3D 0</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,628::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/sbin/tc class show dev enp2s0 classid 0:3 (cwd None)</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,689::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; = <rc> =3D 0</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,690::vsctl::57::root::(commit) Executing commands: = /usr/bin/ovs-vsctl --oneline --format=3Djson -- list Bridge -- list Port = -- list Interface</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,690::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list = 0-7 /usr/bin/ovs-vsctl --oneline --format=3Djson -- list Bridge -- list = Port -- list Interface (cwd</div><div class=3D"">None)</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,720::commands::86::root::(execCmd) SUCCESS: <err> =3D ''; = <rc> =3D 0</div><div = class=3D"">MainProcess|Thread-316590::DEBUG::2017-08-28 = 01:21:56,724::supervdsmServer::99::SuperVdsm.ServerCallback::(wrapper) = return network_caps with {'bridges': {'ovirtmgmt': {'ipv6autoconf': = False, 'addr': '10.9.2.61', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', = 'DNS1': '127.0.0.1', 'IPADDR': '10.9.2.61', 'GATEWAY': '10.9.2.1', = 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', = 'BOOTPROTO': 'none', 'STP': 'off', 'DNS2': '209.193.4.7', 'DEVICE': = 'ovirtmgmt', 'MTU': '1500', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, = 'ipv6addrs': [], 'gateway': '10.9.2.1', 'dhcpv4': False, 'netmask': = '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': = ['10.9.2.61/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eno1'], = 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', = 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', = 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', = 'hello_timer': '104', 'multicast_querier_interval': '25500', 'max_age': = '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': = '0', 'priority': '32768', 'multicast_membership_interval': '26000', = 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.7071bc772964', 'bridge_id': '8000.7071bc772964', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '16704', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'ovirt-4': {'ipv6autoconf': False, 'addr': '10.9.3.61', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'IPADDR': '10.9.3.61', 'MTU': = '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', = 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirt-4', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'gateway': '10.9.3.1', = 'dhcpv4': False, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': = 'off', 'ipv4addrs': ['10.9.3.61/24'], 'mtu': '1500', 'ipv6gateway': = '::', 'ports': ['enp2s0.4'], 'opts': {'multicast_last_member_count': = '2', 'hash_elasticity': '4', 'multicast_query_response_interval': = '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', = 'multicast_startup_query_interval': '3125', 'hello_timer': '102', = 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': = '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': = '32768', 'multicast_membership_interval': '26000', 'root_path_cost': = '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.6805ca4606af', 'bridge_id': '8000.6805ca4606af', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '8307', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'dmz': {'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': = 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': = 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'dmz', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'gateway': '', 'dhcpv4': = False, 'netmask': '', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': [], = 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['vnet0', 'enp4s0.100'], = 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', = 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', = 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', = 'hello_timer': '103', 'multicast_querier_interval': '25500', 'max_age': = '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': = '0', 'priority': '32768', 'multicast_membership_interval': '26000', = 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.001b212f794d', 'bridge_id': '8000.001b212f794d', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '7181', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'ovirt-2': {'ipv6autoconf': False, 'addr': '', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': = 'ovirt-2', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], = 'gateway': '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'stp': = 'off', 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.2'], 'opts': {'multicast_last_member_count': '2', = 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', = 'group_fwd_mask': '0x0', 'multicast_snooping': '1', = 'multicast_startup_query_interval': '3125', 'hello_timer': '2', = 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': = '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': = '32768', 'multicast_membership_interval': '26000', 'root_path_cost': = '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.6805ca4606af', 'bridge_id': '8000.6805ca4606af', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '320', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}, 'ovirt-3': {'ipv6autoconf': False, 'addr': '', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': = 'ovirt-3', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': [], = 'gateway': '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'stp': = 'off', 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.3'], 'opts': {'multicast_last_member_count': '2', = 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', = 'group_fwd_mask': '0x0', 'multicast_snooping': '1', = 'multicast_startup_query_interval': '3125', 'hello_timer': '1', = 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': = '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': = '32768', 'multicast_membership_interval': '26000', 'root_path_cost': = '0', 'root_port': '0', 'multicast_querier': '0', = 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', = 'topology_change': '0', 'hello_time': '200', 'root_id': = '8000.6805ca4606af', 'bridge_id': '8000.6805ca4606af', = 'topology_change_timer': '0', 'ageing_time': '30000', = 'nf_call_ip6tables': '0', 'gc_timer': '14656', 'nf_call_arptables': '0', = 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', = 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': = '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': = '0'}}}, 'bondings': {}, 'nameservers': ['127.0.0.1', '209.193.4.7', = '209.112.128.2'], 'nics': {'enp2s0': {'ipv6gateway': '::', = 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': 'no', 'MTU': = '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp2s0', = 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, = 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': = '68:05:ca:46:06:af', 'speed': 1000, 'gateway': ''}, 'eno1': = {'ipv6gateway': '::', 'ipv6autoconf': False, 'addr': '', 'cfg': = {'BRIDGE': 'ovirtmgmt', 'IPV6INIT': 'no', 'MTU': '1500', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'eno1', 'ONBOOT': = 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', = 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '70:71:bc:77:29:64', = 'speed': 1000, 'gateway': ''}, 'enp4s0': {'ipv6gateway': '::', = 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': 'no', 'MTU': = '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp4s0', = 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, = 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': = '00:1b:21:2f:79:4d', 'speed': 1000, 'gateway': ''}}, 'supportsIPv6': = True, 'vlans': {'enp2s0.4': {'iface': 'enp2s0', 'ipv6autoconf': False, = 'addr': '', 'cfg': {'BRIDGE': 'ovirt-4', 'IPV6INIT': 'no', 'VLAN': = 'yes', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', = 'DEVICE': 'enp2s0.4', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'vlanid': 4, = 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, = 'ipv4addrs': [], 'ipv6gateway': '::', 'gateway': ''}, 'enp4s0.100': = {'iface': 'enp4s0', 'ipv6autoconf': False, 'addr': '', 'cfg': {'BRIDGE': = 'dmz', 'IPV6INIT': 'no', 'VLAN': 'yes', 'MTU': '1500', 'NM_CONTROLLED': = 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp4s0.100', 'ONBOOT': 'yes'}, = 'ipv6addrs': [], 'vlanid': 100, 'mtu': '1500', 'dhcpv4': False, = 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'ipv6gateway': '::', = 'gateway': ''}, 'enp2s0.2': {'iface': 'enp2s0', 'ipv6autoconf': False, = 'addr': '', 'cfg': {'BRIDGE': 'ovirt-2', 'IPV6INIT': 'no', 'VLAN': = 'yes', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', = 'DEVICE': 'enp2s0.2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'vlanid': 2, = 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, = 'ipv4addrs': [], 'ipv6gateway': '::', 'gateway': ''}, 'enp2s0.3': = {'iface': 'enp2s0', 'ipv6autoconf': False, 'addr': '', 'cfg': {'BRIDGE': = 'ovirt-3', 'IPV6INIT': 'no', 'VLAN': 'yes', 'MTU': '1500', = 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'DEVICE': 'enp2s0.3', = 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'vlanid': 3, 'mtu': '1500', 'dhcpv4': = False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'ipv6gateway': = '::', 'gateway': ''}}, 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', = 'ipv6autoconf': False, 'addr': '10.9.2.61', 'cfg': {'IPV6INIT': 'no', = 'DEFROUTE':</div><div class=3D"">'yes', 'DNS1': '127.0.0.1', 'IPADDR': = '10.9.2.61', 'GATEWAY': '10.9.2.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', = 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DNS2': = '209.193.4.7', 'DEVICE': 'ovirtmgmt', 'MTU': '1500', 'TYPE': 'Bridge', = 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', = 'gateway': '10.9.2.1', 'dhcpv4': False, 'netmask': '255.255.255.0', = 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': ['10.9.2.61/24'], 'mtu': = '1500', 'ipv6gateway': '::', 'ports': ['eno1']}, 'ovirt-4': {'iface': = 'ovirt-4', 'ipv6autoconf': False, 'addr': '10.9.3.61', 'cfg': = {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'IPADDR': '10.9.3.61', 'MTU': = '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', = 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirt-4', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': [],</div><div = class=3D"">'switch': 'legacy', 'gateway': '10.9.3.1', 'dhcpv4': False, = 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': = ['10.9.3.61/24'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.4']}, 'ovirt-2': {'iface': 'ovirt-2', 'ipv6autoconf': False, = 'addr': '', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'no', 'MTU': '1500', = 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'STP': 'off', = 'DEVICE': 'ovirt-2', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': = True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': '', 'dhcpv4': = False, 'netmask': '', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': [], = 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['enp2s0.2']}, 'dmz': = {'iface': 'dmz', 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': = 'no', 'DEFROUTE': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': = 'no', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'dmz', 'TYPE': = 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': = 'legacy', 'gateway': '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': = False, 'stp': 'off', 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': = '::', 'ports': ['vnet0', 'enp4s0.100']}, 'ovirt-3': {'iface': 'ovirt-3', = 'ipv6autoconf': False, 'addr': '', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': = 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': = 'none', 'STP': 'off', 'DEVICE': 'ovirt-3', 'TYPE': 'Bridge', 'ONBOOT': = 'yes'}, 'bridged': True, 'ipv6addrs': [], 'switch': 'legacy', 'gateway': = '', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'stp': 'off', = 'ipv4addrs': [], 'mtu': '1500', 'ipv6gateway': '::', 'ports': = ['enp2s0.3']}}}</div><div = class=3D"">MainProcess|jsonrpc.Executor/3::DEBUG::2017-08-28 = 01:21:58,027::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper) = call getHardwareInfo with () {}</div><div = class=3D"">MainProcess|jsonrpc.Executor/3::DEBUG::2017-08-28 = 01:21:58,027::supervdsmServer::99::SuperVdsm.ServerCallback::(wrapper) = return getHardwareInfo with {'systemProductName': '', 'systemUUID': = '20DC6410-4F9F-DF11-9A25-7</div></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""><div class=3D""><div = class=3D"" style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><div class=3D"" = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;"><font face=3D"Menlo" class=3D"" = style=3D"font-size: 12px;"><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty         =                     =            <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary(a)ravnalaska.net</a></div><div class=3D"">Systems Manager =                     =                     =  <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska   =                     =     /\                 =    907-450-7251</div><div class=3D"">5245 Airport Industrial = Road         /  \/\         =     907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska =  99709        /\  /    \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span class=3D"" style=3D"font-size: 12px;">Serving All of = Alaska          /  \/  /\  \ \/\ =   =E2=80=9CLove your neighbor as</span></font><br class=3D"" = style=3D"font-family: Monaco;"><font face=3D"Menlo" class=3D""><span = class=3D"" style=3D"font-size: 12px;">Green, green as far as the eyes = can see             yourself=E2=80=9D Matt = 22:39</span></font><div class=3D"" style=3D"font-family: = Menlo;"></div><font face=3D"Menlo" class=3D"" style=3D"font-size: = 12px;"></font><span class=3D"" style=3D"font-size: 12px;"><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div class=3D""><font face=3D"Menlo" = class=3D"" style=3D"font-size: 12px;"><br = class=3D""></font></div></div><span class=3D"" style=3D"font-size: = 12px;"><br class=3D"Apple-interchange-newline"></span></div><span = class=3D"" style=3D"font-size: 12px;"><br = class=3D"Apple-interchange-newline"></span></div><span class=3D"" = style=3D"font-size: 12px;"><br = class=3D"Apple-interchange-newline"></span></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"></div><br class=3D""><blockquote = type=3D"cite" class=3D"">On Aug 27, 2017, at 11:24 PM, Yaniv Kaul <<a = href=3D"mailto:ykaul@redhat.com" class=3D"">ykaul(a)redhat.com</a>> = wrote:<br class=3D""><br class=3D"">vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof</blockquote></div></body></html>= --Apple-Mail=_BF0F8909-F772-4228-ADA6-BEDEE7C22B79--

1 1

OVN provider's firewalld services deployment during engine setup
by Leon Goldberg 19 Sep '17

19 Sep '17

Hey, We're trying to come up with a way to deploy OVN provider's firewalld services during engine setup. The naive solution of querying the user during customization and then installing during STAGE_PACKAGES fails as firewalld configuration happens prior to it. We thought of a couple of possible solutions we'd like your opinions on (ordered in perceived level of difficulty): 1) Ship/require the packages with ovirt-engine without requiring user input. This essentially couples engine with OVN and disregards a future where OVN doesn't run alongside Engine. 2) Install the packages immediately following user input during customization. A bit hacky and doesn't conceptually fit the stage of customization. 3) Move user input to STAGE_INTERNAL_PACKAGES. Feels more disruptive than #1 to the current otopi flow as STAGE_INTERNAL_PACKAGES is dedicated for packages that are required for otopi itself to operate. Not only this doesn't fit conceptually, it introduces user input to a stage that shouldn't have any. 4) Move firewalld configuration to happen after STAGE_PACKAGES. 5) Refactor to prepare the grounds for OVN/Engine separation. At this point this feels very ambiguous. We don't yet know how will containers be accessed (is ssh guaranteed?) in the future or generally how should a remote installation look like. Any input/questions are appreciated. Thanks, Leon

6 10

New design for the Gerrit UI
by Evgheni Dereveanchin 06 Sep '17

06 Sep '17

Hi everyone, The Infra team is working on customizing the look of Gerrit to make it fit better with other oVirt services. I want to share the result of this effort. Hopefully we can gather some feedback before applying the design to oVirt's instance of Gerrit. Please visit the Staging instance to check it out: https://gerrit-staging.phx.ovirt.org/ The new design is inspired by oVirt's main website. There is a known glitch that the "Loading Gerrit Code Review" message is overlapped by the logo when the UI is loading. If you spot any other issues or have ideas for improvement feel free to respond to this thread or share your feedback in the following JIRA ticket: https://ovirt-jira.atlassian.net/browse/OVIRT-912 -- Regards, Evgheni Dereveanchin

17 28

Slack channel
by Marc Young 06 Sep '17

06 Sep '17

Is there hope for slack over IRC? The problem with IRC is all the connect/disconnect chatter (and offline being a black hole)

7 13

planned Gerrit maintenance
by Evgheni Dereveanchin 03 Sep '17

03 Sep '17

Hi everyone, I will be performing updates on gerrit.ovirt.org during the next two hours. Within this period the Gerrit UI and Git repositories may be unavailable. I will follow up as soon as maintenance activities are over. -- Regards, Evgheni Dereveanchin

5 5

Re: [ovirt-devel] vdsm vds.dispatcher
by Piotr Kliczewski 31 Aug '17

31 Aug '17

This change was backported to 4.1 and you can find it in 4.19.7+. This log entry appears in the logs when a client disconnects. 3 seconds is much too often and the only client could do it is mom (adding Martin to confirm). On Mon, Aug 28, 2017 at 12:11 AM, Gary Pedretty <gary(a)ravnalaska.net> wrote: > vdsm-4.18.21-1.el7.centos.x86_64 > > the entries are about every 3 seconds on all hosts > > > ------------------------------------------------------------------------ > Gary Pedretty gary(a)ravnalaska.net > Systems Manager www.flyravn.com > Ravn Alaska /\ 907-450-7251 > 5245 Airport Industrial Road / \/\ 907-450-7238 fax > Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment > Serving All of Alaska / \/ /\ \ \/\ “Love your neighbor as > Green, green as far as the eyes can see yourself” Matt 22:39 > ------------------------------------------------------------------------ > > > > > On Aug 27, 2017, at 12:46 PM, Piotr Kliczewski <piotr.kliczewski(a)gmail.com> > wrote: > > Hello, > > This is not an error and it was modified to reduce logging level. > Which vdsm version are you using? > > Thanks, > Piotr > >

3 4

blog post: quick(er) development / testing of GWT code using SDM
by Greg Sheremeta 31 Aug '17

31 Aug '17

Hi, I wrote a blog post detailing how to use GWT SDM to let you change GWT code and see the results in a few seconds. http://www.ovirt.org/blog/2017/08/ovirt-webadmin-gwt-debug-quick-refresh/ Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

2 1

[ OST Failure Report ] [ oVirt master ] [ 2017-08-30 ] [add_hosts]
by Barak Korren 31 Aug '17

31 Aug '17

Test failed: [ 002_bootstrap.add_hosts ] Link to suspected patches: We suspect this is due to change to LLDPAD in upstream CentOS repos. We can't tell the exact point it was introduced because other ovirt-engine regressions introduced too much noise into the system. It also seems that the failure is not 100% reproducible sine we have runs that do not encounter it. Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2151 (This job checked a specific patch to ovirt-log-collector but failure seems unrelated, and also happens on vanille 'tested' repo right now). Link to all logs: VDSM logs seem to be most relevant here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2151/artifact… Error snippet from log: - From VDSM logs (note it exists very quickly): <error> 2017-08-30 05:55:52,208-0400 INFO (MainThread) [vds] (PID: 4594) I am the actual vdsm 4.20.2-133.git0ce4485.el7.centos lago-basic-suite-master-host-0 (3.10.0-514.2.2.el7.x86_64) (vdsmd:148) 2017-08-30 05:55:52,209-0400 INFO (MainThread) [vds] VDSM will run with cpu affinity: frozenset([1]) (vdsmd:254) 2017-08-30 05:55:52,253-0400 INFO (MainThread) [storage.check] Starting check service (check:92) 2017-08-30 05:55:52,257-0400 INFO (MainThread) [storage.Dispatcher] Starting StorageDispatcher... (dispatcher:48) 2017-08-30 05:55:52,257-0400 INFO (check/loop) [storage.asyncevent] Starting <EventLoop running=True closed=False at 0x45317520> (asyncevent:125) 2017-08-30 05:55:52,288-0400 INFO (MainThread) [vdsm.api] START registerDomainStateChangeCallback(callbackFunc=<functools.partial object at 0x2b47b50>) from=internal, task_id=2cebe9ef-358e-47d7-81a6-8c4b54b9cd6d (api:46) 2017-08-30 05:55:52,288-0400 INFO (MainThread) [vdsm.api] FINISH registerDomainStateChangeCallback return=None from=internal, task_id=2cebe9ef-358e-47d7-81a6-8c4b54b9cd6d (api:52) 2017-08-30 05:55:52,288-0400 INFO (MainThread) [MOM] Preparing MOM interface (momIF:53) 2017-08-30 05:55:52,289-0400 INFO (MainThread) [MOM] Using named unix socket /var/run/vdsm/mom-vdsm.sock (momIF:62) 2017-08-30 05:55:52,289-0400 INFO (MainThread) [root] Unregistering all secrets (secret:91) 2017-08-30 05:55:52,307-0400 INFO (MainThread) [vds] Setting channels' timeout to 30 seconds. (vmchannels:221) 2017-08-30 05:55:52,309-0400 INFO (vmrecovery) [vds] recovery: completed in 0s (clientIF:516) 2017-08-30 05:55:52,310-0400 INFO (MainThread) [vds.MultiProtocolAcceptor] Listening at :::54321 (protocoldetector:196) 2017-08-30 05:55:52,496-0400 INFO (http) [vds] Server running (http:58) 2017-08-30 05:55:52,742-0400 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=85768015-8ecb-48e3-9307-f671bfc33c65 (api:46) 2017-08-30 05:55:52,743-0400 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=85768015-8ecb-48e3-9307-f671bfc33c65 (api:52) 2017-08-30 05:55:52,743-0400 WARN (periodic/1) [throttled] MOM not available. (throttledlog:103) 2017-08-30 05:55:52,744-0400 WARN (periodic/1) [throttled] MOM not available, KSM stats will be missing. (throttledlog:103) 2017-08-30 05:55:55,043-0400 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:67) 2017-08-30 05:55:55,045-0400 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:759) 2017-08-30 05:55:55,049-0400 INFO (MainThread) [vds] Stopping http server (http:79) 2017-08-30 05:55:55,049-0400 INFO (http) [vds] Server stopped (http:69) 2017-08-30 05:55:55,050-0400 INFO (MainThread) [root] Unregistering all secrets (secret:91) 2017-08-30 05:55:55,052-0400 INFO (MainThread) [vdsm.api] START prepareForShutdown(options=None) from=internal, task_id=ffde5caa-fa44-49ab-bdd1-df81519680a3 (api:46) 2017-08-30 05:55:55,089-0400 INFO (MainThread) [storage.Monitor] Shutting down domain monitors (monitor:222) 2017-08-30 05:55:55,090-0400 INFO (MainThread) [storage.check] Stopping check service (check:105) 2017-08-30 05:55:55,090-0400 INFO (check/loop) [storage.asyncevent] Stopping <EventLoop running=True closed=False at 0x45317520> (asyncevent:220) 2017-08-30 05:55:55,090-0400 INFO (MainThread) [vdsm.api] FINISH prepareForShutdown return=None from=internal, task_id=ffde5caa-fa44-49ab-bdd1-df81519680a3 (api:52) 2017-08-30 05:55:55,091-0400 INFO (MainThread) [vds] Stopping threads (vdsmd:159) 2017-08-30 05:55:55,091-0400 INFO (MainThread) [vds] Exiting (vdsmd:170) </error> - From SuperVDSM logs: <error> MainThread::ERROR::2017-08-30 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command timed out.\n", '', 'eth1') MainThread::DEBUG::2017-08-30 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool get-lldp -i eth0 adminStatus (cwd None) </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

7 16

master OST is now green. Thanks everyone who helped!
by Barak Korren 31 Aug '17

31 Aug '17

The following OST run, that included 82 different patches and ~40 engine patches among them was successful: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2208/ This means that engine master is not longer broken, and there are no other issues that affect OST. Thanks everyone who worked hard on this! You are free to resume merging any engine patches you may have refrained from merging so far. Thanks, Barak. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

GSoC final evaluation
by Valentina Makarova 31 Aug '17

31 Aug '17

Hello oVirt Team! I'm GSoC participant and yesterday was last day of GSoC 2017 program. Thank you very much for help, answers on my questions, great project and interesting experience! There is my report about my work in ovirt-system-tests https://gist.github.com/vmakarova/62c2941d4723bc2673427db22cf72a29 Good luck! Sincerely, Valentina Makarova

3 2

Please note: ovirt-hosted-engine-ha builds are failing
by Barak Korren 31 Aug '17

31 Aug '17

Hi, Builds of ovirt-hosted-engine-ha had been failing for the last 3 merged patches. This is currently not being reported on gerrit because of a known issue we are working to fix [1]. The 1st patch where builds started failing is: https://gerrit.ovirt.org/#/c/81224/ Failing build jobs for this patch are the following: - http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifacts-… - http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifacts-… Build logs for failing jobs are respectively at these locations: - http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifacts-… - http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifacts-… [1]: https://ovirt-jira.atlassian.net/browse/OVIRT-1502 -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 2

Re: [ovirt-devel] URGENT! OST broken! please stop merging patches to ovirt-engine!
by Greg Sheremeta 30 Aug '17

30 Aug '17

On Aug 30, 2017 8:46 AM, "Maor Lipchuk" <mlipchuk(a)redhat.com> wrote: Yes, there is already an acked patch that solves it. The weird thing is that the OST ran successfully when running with "ci please build" although it seems that the artifact was not added to it. That doesn't run full OST. It only runs build-artifacts :) On Wed, Aug 30, 2017 at 3:17 PM, Greg Sheremeta <gshereme(a)redhat.com> wrote: > > On Wed, Aug 30, 2017 at 7:13 AM, Barak Korren <bkorren(a)redhat.com> wrote: >> >> We have two unresolved issues that are currently causing all master >> OST runs to fail and preventing us from effectively finding >> regressions. >> >> The 1st issue, which was already reported on Sunday, is a regression >> that is causing vm_run to fail and was probably introduced by this >> patch: >> https://gerrit.ovirt.org/#/c/79033/41 > > > cc'd Shubham and Maor. Is anyone looking into this? > >> >> >> >> Here is a recent run that failed with this: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/ >> >> Logs are here: >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/2147/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-002_bootstrap.py/ >> >> The 2nd issue is a new one, that seems to be causing add-host failures >> and I've just send another email about it. The gist of this is that >> I'm siing the following failures in the supervdsm logs, and vdsm is >> not loading: >> >> MainThread::ERROR::2017-08-30 >> 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable >> LLDP on eth1 >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", >> line 51, in _lldp_init >> Lldp.enable_lldp_on_iface(device) >> File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", >> line 30, in enable_lldp_on_iface >> lldptool.enable_lldp_on_iface(iface, rx_only) >> File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/ lldptool.py", >> line 46, in enable_lldp_on_iface >> raise EnableLldpError(rc, out, err, iface) >> EnableLldpError: (1, >> "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command >> timed out.\n", '', 'eth1') >> MainThread::DEBUG::2017-08-30 >> 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool >> get-lldp -i eth0 adminStatus (cwd None) >> >> >> This failure does not seem to have been introduced by the platform and >> not oVirt code, because it also happens on code that already passed >> OST a few days ago. >> >> Please avoid merging any patches except to the purpose of resolving >> these issues. Nothing is making it into the 'tested' and nightly >> snapshot repos anyway ATM. >> >> -- >> Barak Korren >> RHV DevOps team , RHCE, RHCi >> Red Hat EMEA >> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel > >

2 1

URGENT! OST broken! please stop merging patches to ovirt-engine!
by Barak Korren 30 Aug '17

30 Aug '17

We have two unresolved issues that are currently causing all master OST runs to fail and preventing us from effectively finding regressions. The 1st issue, which was already reported on Sunday, is a regression that is causing vm_run to fail and was probably introduced by this patch: https://gerrit.ovirt.org/#/c/79033/41 Here is a recent run that failed with this: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/ Logs are here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/artifact… The 2nd issue is a new one, that seems to be causing add-host failures and I've just send another email about it. The gist of this is that I'm siing the following failures in the supervdsm logs, and vdsm is not loading: MainThread::ERROR::2017-08-30 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command timed out.\n", '', 'eth1') MainThread::DEBUG::2017-08-30 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool get-lldp -i eth0 adminStatus (cwd None) This failure does not seem to have been introduced by the platform and not oVirt code, because it also happens on code that already passed OST a few days ago. Please avoid merging any patches except to the purpose of resolving these issues. Nothing is making it into the 'tested' and nightly snapshot repos anyway ATM. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

4 3

[ oVirt Devel ] [oVirt master ] [ OST Failure Report ] [ basic_suite_master & upgrade_suites failed ]
by Daniel Belenky 30 Aug '17

30 Aug '17

*Test Failed:* 1. *basic_suite: *002_bootstrap.add_dc and 2. *upgrade_from_release: *001_upgrade_engine.test_initialize_engine 3. *upgrade_from_prevrelease*: *Link to test logs:* 1. *basic suite logs <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2136/artifact…>* 2. *upgrade from prev release suite logs <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2136/artifact…>* 3. *upgrade from release suite logs <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2136/artifact…>* *Suspected patch: https://gerrit.ovirt.org/#/c/79033/41 <https://gerrit.ovirt.org/#/c/79033/41>* *Please note that this change was the only change under those tests.* *Error snippet from basic-suite /var/log/ovirt-engine/server.log:* ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /ovirt-engine/api/v4/datacenters: java.lang.RuntimeException: org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: org/ovirt/api/metamodel/server/ValidationException at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:245) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:180) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:284) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141] Caused by: org.jboss.resteasy.spi.UnhandledException: java.lang.NoClassDefFoundError: org/ovirt/api/metamodel/server/ValidationException at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:78) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:222) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:179) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:422) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) [undertow-servlet-1.4.18.Final.jar:1.4.18.Final] ... 73 more Caused by: java.lang.NoClassDefFoundError: org/ovirt/api/metamodel/server/ValidationException at org.ovirt.engine.api.resource.DataCentersResource.doAdd(DataCentersResource.java:58) [restapi-definition.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] ... 88 more Caused by: java.lang.ClassNotFoundException: org.ovirt.api.metamodel.server.ValidationException from [Module "org.ovirt.engine.api.restapi-definition" from local module loader @61064425 (finder: local module finder @7b1d7fff (roots: /usr/share/ovirt-engine-wildfly-overlay/modules,/usr/share/ovirt-engine/modules/common,/usr/share/ovirt-engine-extension-aaa-jdbc/modules,/usr/share/ovirt-engine-extension-aaa-ldap/modules,/usr/share/ovirt-engine-wildfly/modules,/usr/share/ovirt-engine-wildfly/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198) [jboss-modules.jar:1.6.0.CR2] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:412) [jboss-modules.jar:1.6.0.CR2] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:400) [jboss-modules.jar:1.6.0.CR2] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116) [jboss-modules.jar:1.6.0.CR2] ... 99 more *Error snippet from both upgrade suites (engine setup output):*[ ERROR ] Failed to execute stage 'Closing up': Command '/bin/firewall-cmd' failed to execute -- DANIEL BELENKY RHV DEVOPS Red Hat EMEA <https://www.redhat.com/> IRC: #rhev-integ #rhev-dev <https://red.ht/sig>

3 3

vdsm fails with: [Errno 2] No such file or directory: '/var/run/vdsm/svdsm.sock'
by Marcin Mirecki 29 Aug '17

29 Aug '17

Hi, I'm trying to add a add a new host, but each time the installation fails due to service supervdsm failing with: Aug 29 11:33:23 f25_55 daemonAdapter[1166]: Traceback (most recent call last): Aug 29 11:33:23 f25_55 daemonAdapter[1166]: File "/usr/lib64/python2.7/multiprocessing/util.py", line 274, in _run_finalizers Aug 29 11:33:23 f25_55 daemonAdapter[1166]: finalizer() Aug 29 11:33:23 f25_55 daemonAdapter[1166]: File "/usr/lib64/python2.7/multiprocessing/util.py", line 207, in __call__ Aug 29 11:33:23 f25_55 daemonAdapter[1166]: res = self._callback(*self._args, **self._kwargs) Aug 29 11:33:23 f25_55 daemonAdapter[1166]: OSError: [Errno 2] No such file or directory: '/var/run/vdsm/svdsm.sock' Aug 29 11:33:23 f25_55 systemd[1]: supervdsmd.service: Main process exited, code=exited, status=1/FAILURE Aug 29 11:33:23 f25_55 systemd[1]: supervdsmd.service: Unit entered failed state. I tried it on different freshly intalled f25 hosts. Tried installing vdsm from engine and manually, the problem occurs every time. Thanks, Marcin

2 2

Re: [ovirt-devel] CI failure with tox
by Barak Korren 29 Aug '17

29 Aug '17

On 29 August 2017 at 11:28, Francesco Romani <fromani(a)redhat.com> wrote: > Hi Infra, > > > while running CI on some patches (example: > https://gerrit.ovirt.org/#/c/80967/) I see this failure[1] > > It seems to me something on the CI worker is broken, could anyone check? > > let me stress that the tests pass on my fedora laptop and that the > failure seems unrelated to my patch(es). Note that its failing inside the EL7 chroot but not the FC25 chroot. Could you be using some version of a library call that does not exist on EL7 or returns different output? Not that test scripts always run inside an isolated chroot, so chances of something in the build slave affecting them are slim as long as you're not trying to connect to any external services or devices. This does not look like an obvious build slave issue, so CCing devel. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

[ OST Failure Report ] [ oVirt master ] [ 20-08-2017 ] [add_cluster]
by Barak Korren 29 Aug '17

29 Aug '17

Test failed: [ 002_bootstrap.add_cluster ] Link to suspected patches: https://gerrit.ovirt.org/#/c/80692/7 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2066 Link to all logs: Error snippet from log: <error> Fault reason is "Operation Failed". Fault detail is "javax.ejb.EJBException: java.lang.NullPointerException at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:214) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:266) at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) ... at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.ovirt.engine.core.bll.Backend.evaluateCorrelationId(Backend.java:506) at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:492) at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:481) at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:434) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ... </error> ********************* Please note that all OST tests including recent engine patches had been broken since a patch perviously reported. We report this because we've noticed a change in the failing test. Note that no new engine-patches are being accepted into the 'tested' repositories, and this is expected to continue until the existing issues are fixed. ********************* -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 1

[ OST Failure Report ] [ oVirt master ] [ 28/08/2017 ] [vm_run]
by Barak Korren 28 Aug '17

28 Aug '17

Test failed: [ basic_sanity.vm_run ] Link to suspected patches: https://gerrit.ovirt.org/#/c/79033/41 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2034 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2034/artifact… Error snippet from log: <error> RequestError: status: 500 reason: Internal Server Error detail: General command validation failure. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

bugs targeted to "pre-alpha" release
by Sandro Bonazzola 28 Aug '17

28 Aug '17

Hi, not sure who added "pre-alpha" release to bugzilla, but "pre-alpha" is not a release so it's meaningless. Please note we have 52 bugs on engine with that release set: https://bugzilla.redhat.com/buglist.cgi?target_release=pre-alpha&product=ov… Please ensure your bugs have a valid target release (in this case I guess 4.2.0) so we can get rid of that meaningless release target. Thanks, -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

3 3

ImageIO issues
by Piotr Kliczewski 28 Aug '17

28 Aug '17

Hi, For some time I am attempting to upload 14GB image over my local network. I faced network service crash on my vdsm host (it runs imageio daemon). Now I got following issue: - daemon side: 2017-08-28 12:02:01,289 ERROR (Thread-194) [web] 192.168.1.107 - PUT /db08fc46-76f3-448a-8899-acd14a7cda41 500 210 (263.16s) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 48, in __call__ resp = self.dispatch(request) File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/web.py", line 73, in dispatch return method(*match.groups()) File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 162, in put op.run() File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 68, in run self._run() File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 156, in _run self._receive_chunk(dst, buf, count) File "/usr/lib/python2.7/site-packages/ovirt_imageio_common/directio.py", line 192, in _receive_chunk raise errors.PartialContent(self.size, self.done) PartialContent: Requested 8388608 bytes, available 6389760 bytes - proxy side: 192.168.1.107 - - [28/Aug/2017 12:01:56] "PUT /images/db08fc46-76f3-448a-8899-acd14a7cda41 HTTP/1.1" 500 59 ---------------------------------------- Exception happened during processing of request from ('192.168.1.107', 51716) Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 596, in process_request_thread self.finish_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 331, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.7/SocketServer.py", line 654, in __init__ self.finish() File "/usr/lib64/python2.7/SocketServer.py", line 713, in finish self.wfile.close() File "/usr/lib64/python2.7/socket.py", line 283, in close self.flush() File "/usr/lib64/python2.7/socket.py", line 307, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib64/python2.7/ssl.py", line 753, in sendall v = self.send(data[count:]) File "/usr/lib64/python2.7/ssl.py", line 719, in send v = self._sslobj.write(data) error: [Errno 32] Broken pipe ---------------------------------------- I am using sources from couple of days ago and running both services on fedora 25. Is it known issue or should I open BZ for it? Thanks, Piotr

2 6

Memory stats in VDSM
by Tomáš Golembiovský 28 Aug '17

28 Aug '17

Hi, there is ongoing effort to change how VDSM collects information about memory usage from guests. We used to use oVirt guest agent to get the statistics about free memory, swap usage, etc. This is going to change and we will use stats provided by VirtIO Balloon driver. [1] This should not break MOM nor Engine (current or previous versions). There is however small downside to this change. We will not be able to fetch information about buffers and caches on Linux guests anymore (mem_cached and mem_buffers in memoryStats). The corollary of this is that free memory reported by Engine will also no longer include those statistics and reported free memory will be really (only) free memory. I now there will be mixed feelings about this and we would like to get the information about buffers and caches back to the VDSM. Correctly via balloon driver statistics. [2] This effort will however take some time. Tomas [1] https://gerrit.ovirt.org/#/q/topic:memory-stats [2] https://lists.nongnu.org/archive/html/qemu-devel/2017-08/msg05239.html -- Tomáš Golembiovský <tgolembi(a)redhat.com>

3 6

VDSM failed building on PPC64LE
by Barak Korren 28 Aug '17

28 Aug '17

Please note that the following VDSM patch failed building on PPC64LE: https://gerrit.ovirt.org/#/c/75784 He is the failed job: http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/2384/ Due to OVIRT-1502 erro was not show in Gerrit, please note that when this happens is seems to always mean that one of the post-merge jobs failed. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

3 3

[oVirt-engine-SDK][Go SDK]
by Joey Ma 28 Aug '17

28 Aug '17

Hi all, I've made a Go sdk for oVirt-engine v4.0+ implemented. The Go sdk generator locates (mainly Java) in imjoey/ovirt-engine-sdk-go of github, while for the Go sdk itself is in imjoey/go-ovirt. With the help of some dear oVirt sdk maintainers, the project has achieved nearly 90 percent now. It is used in production on my job. Meanwhile It's still under active development and the APIs are not stable currently. If it's useful for you, I'm very appreciated for your feedbacks. The project is still on the way, so welcome to contribute, both issues and PRs. That is my pleasure. Glad to meet you all. Best regards, Joey Ma

2 2

ovirt-provider-ovn_master_build-artifacts-el7-x86_64 Failing Due to Spec Problem
by Anton Marchukov 28 Aug '17

28 Aug '17

Hello All. Just noted that change queue job [1] for ovirt-provider-ovn started to fail consistently while the actual reason for that seems to be RPM build failure at [2]: *14:47:42* Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/jenkins/workspace/ovirt-provider-ovn_master_build-artifacts-el7-x86_64/ovirt-provider-ovn/rpmbuild/BUILDROOT/ovirt-provider-ovn-1.1-2.20170824144740.gitcf66669.el7.centos.x86_64*14:47:42* error: Installed (but unpackaged) file(s) found:*14:47:42* /usr/share/doc/ovirt-provider-ovn/README.adoc*14:47:42* *14:47:42* *14:47:42* RPM build errors:*14:47:42* Installed (but unpackaged) file(s) found:*14:47:42* /usr/share/doc/ovirt-provider-ovn/README.adoc*14:47:42* make: *** [rpm] Error 1*14:47:42* Took 2 seconds Possible root cause for that is [3] (Install README). However, I see CI passed there within the change at least before the rebase. [1] http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1996/console [2] http://jenkins.ovirt.org/job/ovirt-provider-ovn_master_build-artifacts-el7-… [3] https://gerrit.ovirt.org/#/c/80801/ -- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

4 3

vdsm vds.dispatcher
by Gary Pedretty 28 Aug '17

28 Aug '17

--Apple-Mail=_36946AB3-C9DF-47C3-AC25-48E380106247 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Why on even a new clean install of a glusterized self-hosted engine = setup do all the hosts fill the messages log with=20 journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof An entry every few seconds even with no VMs setup and just the hosted = engine running. I know it is listed as a bug on redhat bugzilla page, = but is this going to be normal until version 4.2 comes out? Gary ------------------------------------------------------------------------ Gary Pedretty gary(a)ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Green, green as far as the eyes can see yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ --Apple-Mail=_36946AB3-C9DF-47C3-AC25-48E380106247 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><span class=3D"" style=3D"font-family: LucidaGrande;">Why on = even a new clean install of a glusterized self-hosted engine setup do = all the hosts fill the messages log with </span><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;">journal: vdsm vds.dispatcher ERROR = SSL error during reading data: unexpected eof</div><div class=3D"" = style=3D"font-family: LucidaGrande;"><div class=3D"">journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div></div><div class=3D"" style=3D"font-family: LucidaGrande;"><div = class=3D"">journal: vdsm vds.dispatcher ERROR SSL error during reading = data: unexpected eof</div></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><div class=3D"">journal: vdsm vds.dispatcher ERROR SSL = error during reading data: unexpected eof</div></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><div class=3D"">journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div></div><div class=3D"" style=3D"font-family: LucidaGrande;"><br = class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">An entry every few seconds even with no VMs setup and = just the hosted engine running.  I know it is listed as a bug on = redhat bugzilla page, but is this going to be normal until version 4.2 = comes out?</div><div class=3D"" style=3D"font-family: LucidaGrande;"><br = class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">Gary</div><div class=3D""><br class=3D""></div><div = class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div = style=3D"text-align: start; text-indent: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"text-align: start; text-indent: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div = style=3D"text-align: start; text-indent: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"text-align: start; text-indent: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty         =                     =            <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary(a)ravnalaska.net</a></div><div class=3D"">Systems Manager =                     =                     =  <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska   =                     =     /\                 =    907-450-7251</div><div class=3D"">5245 Airport Industrial = Road         /  \/\         =     907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska =  99709        /\  /    \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska          /  \/  /\  \ \/\ =   =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Green, = green as far as the eyes can see           =   yourself=E2=80=9D Matt 22:39</span></font><div style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: = Menlo;" class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""></body></html>= --Apple-Mail=_36946AB3-C9DF-47C3-AC25-48E380106247--

3 2