<div dir="ltr"><div><br></div><div>Hi, I realised that gluster 3.7.8 was released for GA. So I updated manually using "yum -y install glusterfs*" gluster get's updated normally, but unfortunately things stopped to work completely. ovirt_hosted_engine_ha was not able to connect to my gluster storage domain (as you can see below). I'm not using ovirt + gluster in hyperconverged way. I have 3 external gluster hosts. </div><div><br></div><div>As a workaround I've executed "<span class="s1"><b>yum</b></span><span class="s2"> downgrade glusterfs*" and after doing this ovirt started to work again...but as a collateral effect now I can't migrate my vm's anymore. On the vdsm logs I can see the following errors when trying to manually migrate a any VM.... Anyone knows how to fix it? </span></div><div><span class="s2"><br></span></div><div><span class="s2">
<p class="p1"><span class="s1">Thread-66055::DEBUG::2016-02-17 11:05:57,497::migration::453::virt.vm::(stop) vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::stopping migration downtime thread</span></p>
<p class="p1"><span class="s1">Thread-66055::ERROR::2016-02-17 11:05:57,497::migration::208::virt.vm::(_recover) vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::unable to connect to server at '<a href="http://kvm2.brightsquid.com:49152">kvm2.brightsquid.com:49152</a>': No route to host</span></p>
<p class="p1"><span class="s1">Thread-66055::DEBUG::2016-02-17 11:05:57,497::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response</span></p>
<p class="p1"><span class="s1">Thread-66056::DEBUG::2016-02-17 11:05:57,498::migration::450::virt.vm::(run) vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::migration downtime thread exiting</span></p>
<p class="p1"><span class="s1">Thread-66055::DEBUG::2016-02-17 11:05:57,540::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4327804740, "05681896-a76a-4ae1-879e-8fe5d28634e1": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|05681896-a76a-4ae1-879e-8fe5d28634e1"}</span></p>
<p class="p1"><span class="s1">Thread-66055::ERROR::2016-02-17 11:05:57,541::migration::310::virt.vm::(run) vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::</span><span class="s2">Failed</span><span class="s1"> to migrate</span></p>
<p class="p1"><span class="s1">Traceback (most recent call last):</span></p>
<p class="p1"><span class="s1"> File "/usr/share/vdsm/virt/migration.py", line 294, in run</span></p>
<p class="p1"><span class="s1"> self._startUnderlyingMigration(time.time())</span></p>
<p class="p1"><span class="s1"> File "/usr/share/vdsm/virt/migration.py", line 364, in _startUnderlyingMigration</span></p>
<p class="p1"><span class="s1"> self._perform_migration(duri, muri)</span></p>
<p class="p1"><span class="s1"> File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration</span></p>
<p class="p1"><span class="s1"> self._vm._dom.migrateToURI3(duri, params, flags)</span></p>
<p class="p1"><span class="s1"> File "/usr/share/vdsm/virt/virdomain.py", line 68, in f</span></p>
<p class="p1"><span class="s1"> ret = attr(*args, **kwargs)</span></p>
<p class="p1"><span class="s1"> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper</span></p>
<p class="p1"><span class="s1"> ret = f(*args, **kwargs)</span></p>
<p class="p1"><span class="s1"> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3</span></p>
<p class="p1"><span class="s1"> if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)</span></p>
<p class="p1"><span class="s1">libvirtError: unable to connect to server at '<a href="http://kvm2.brightsquid.com:49152">kvm2.brightsquid.com:49152</a>': No route to host</span></p>
<p class="p1"><span class="s1">Thread-68::DEBUG::2016-02-17 11:05:57,652::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-39 /usr/bin/dd if=/rhev/data-center/mnt/gluster2.brightsquid.com:_home_export_iso/61827b7b-e255-44f5-a791-482a144be29f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None)</span></p>
<p class="p1"><span class="s1">Thread-68::DEBUG::2016-02-17 11:05:57,662::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n341 bytes (341 B) copied, 0.000713967 s, 478 kB/s\n'; <rc> = 0</span></p><p class="p1"><span class="s1"><br></span></p></span></div>
<div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Issue to connect to the gluster storage.</div><div>*****************************************************************************</div><div>Feb 17 01:25:40 kvm2 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server</div><div>Feb 17 01:25:40 kvm2 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server</div><div>Feb 17 01:25:40 kvm2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to storage server failed' - trying to restart agent</div><div>Feb 17 01:25:40 kvm2 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to storage server failed' - trying to restart agent</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established</div><div>Feb 17 01:25:41 kvm2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=glusterfs sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle#012 data)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch#012 .set_storage_domain(client, sd_type, **options)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain#012 self._backends[client].connect()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 456, in connect#012 self._dom_type)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 108, in get_domain_path#012 " in {1}".format(sd_uuid, parent))#012BackendFailureException: path to storage domain 7d376952-312b-4539-b809-a8fa740f7883 not found in /rhev/data-center/mnt/glusterSD</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: ERROR:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=glusterfs sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: Traceback (most recent call last):</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: data)</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: .set_storage_domain(client, sd_type, **options)</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: self._backends[client].connect()</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 456, in connect</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: self._dom_type)</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 108, in get_domain_path</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: " in {1}".format(sd_uuid, parent))</div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: BackendFailureException: path to storage domain 7d376952-312b-4539-b809-a8fa740f7883 not found in /rhev/data-center/mnt/glusterSD</div><div>Feb 17 01:25:41 kvm2 journal: vdsm vds ERROR failed to retrieve Hosted Engine HA info#012Traceback (most recent call last):#012 File "/usr/share/vdsm/API.py", line 1842, in _getHaInfo#012 stats = instance.get_all_stats()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats#012 self._configure_broker_conn(broker)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn#012 dom_type=dom_type)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain#012 .format(sd_type, options, e))#012RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'glusterfs', 'sd_uuid': '7d376952-312b-4539-b809-a8fa740f7883'}: Request failed: <class 'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'></div><div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed</div><div>^C</div><div><br></div></div>