4:13 p.m.
This is a multi-part message in MIME format.
--------------060807010707030303030502
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
On 02/17/2016 04:57 PM, Luiz Claudio Prazeres Goncalves wrote:
Hi, I realised that gluster 3.7.8 was released for GA. So I updated
manually using "yum -y install glusterfs*" gluster get's updated
normally, but unfortunately things stopped to work completely.
ovirt_hosted_engine_ha was not able to connect to my gluster storage
domain (as you can see below). I'm not using ovirt + gluster in
hyperconverged way. I have 3 external gluster hosts.
What does the mount log on the hypervisor say? Could you attach that?
As a workaround I've executed "*yum*downgrade glusterfs*" and after
doing this ovirt started to work again...but as a collateral effect
now I can't migrate my vm's anymore. On the vdsm logs I can see the
following errors when trying to manually migrate a any VM.... Anyone
knows how to fix it?
Thread-66055::DEBUG::2016-02-17
11:05:57,497::migration::453::virt.vm::(stop)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::stopping migration
downtime thread
Thread-66055::ERROR::2016-02-17
11:05:57,497::migration::208::virt.vm::(_recover)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::unable to connect to
server at 'kvm2.brightsquid.com:49152
<http://kvm2.brightsquid.com:49152>';: No route to host
Thread-66055::DEBUG::2016-02-17
11:05:57,497::stompreactor::389::jsonrpc.AsyncoreClient::(send)
Sending response
Thread-66056::DEBUG::2016-02-17
11:05:57,498::migration::450::virt.vm::(run)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::migration downtime thread
exiting
Thread-66055::DEBUG::2016-02-17
11:05:57,540::__init__::206::jsonrpc.Notification::(emit) Sending
event {"params": {"notify_time": 4327804740,
"05681896-a76a-4ae1-879e-8fe5d28634e1": {"status": "Migration
Source"}}, "jsonrpc": "2.0", "method":
"|virt|VM_status|05681896-a76a-4ae1-879e-8fe5d28634e1"}
Thread-66055::ERROR::2016-02-17
11:05:57,541::migration::310::virt.vm::(run)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::Failedto migrate
Traceback (most recent call last):
File "/usr/share/vdsm/virt/migration.py", line 294, in run
self._startUnderlyingMigration(time.time())
File "/usr/share/vdsm/virt/migration.py", line 364, in
_startUnderlyingMigration
self._perform_migration(duri, muri)
File "/usr/share/vdsm/virt/migration.py", line 403, in
_perform_migration
self._vm._dom.migrateToURI3(duri, params, flags)
File "/usr/share/vdsm/virt/virdomain.py", line 68, in f
ret = attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 124, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in
migrateToURI3
if ret == -1: raise libvirtError ('virDomainMigrateToURI3()
failed', dom=self)
libvirtError: unable to connect to server at
'kvm2.brightsquid.com:49152 <http://kvm2.brightsquid.com:49152>';: No
route to host
Thread-68::DEBUG::2016-02-17
11:05:57,652::fileSD::173::Storage.Misc.excCmd::(getReadDelay)
/usr/bin/taskset --cpu-list 0-39 /usr/bin/dd
if=/rhev/data-center/mnt/gluster2.brightsquid.com:_home_export_iso/61827b7b-e255-44f5-a791-482a144be29f/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)
Thread-68::DEBUG::2016-02-17
11:05:57,662::fileSD::173::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: <err> = '0+1 records in\n0+1 records out\n341 bytes (341 B)
copied, 0.000713967 s, 478 kB/s\n'; <rc> = 0
Issue to connect to the gluster storage.
*****************************************************************************
Feb 17 01:25:40 kvm2 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage
server
Feb 17 01:25:40 kvm2 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage
server
Feb 17 01:25:40 kvm2 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to
storage server failed' - trying to restart agent
Feb 17 01:25:40 kvm2 ovirt-ha-agent:
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to
storage server failed' - trying to restart agent
Feb 17 01:25:41 kvm2 ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
established
Feb 17 01:25:41 kvm2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error
handling request, data: 'set-storage-domain FilesystemBackend
dom_type=glusterfs
sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'#012Traceback (most
recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 166, in handle#012 data)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 299, in _dispatch#012 .set_storage_domain(client, sd_type,
**options)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 66, in set_storage_domain#012
self._backends[client].connect()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 456, in connect#012 self._dom_type)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 108, in get_domain_path#012 " in {1}".format(sd_uuid,
parent))#012BackendFailureException: path to storage domain
7d376952-312b-4539-b809-a8fa740f7883 not found in
/rhev/data-center/mnt/glusterSD
Feb 17 01:25:41 kvm2 ovirt-ha-broker:
ERROR:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Error
handling request, data: 'set-storage-domain FilesystemBackend
dom_type=glusterfs sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'
Feb 17 01:25:41 kvm2 ovirt-ha-broker: Traceback (most recent call last):
Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 166, in handle
Feb 17 01:25:41 kvm2 ovirt-ha-broker: data)
Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 299, in _dispatch
Feb 17 01:25:41 kvm2 ovirt-ha-broker: .set_storage_domain(client,
sd_type, **options)
Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 66, in set_storage_domain
Feb 17 01:25:41 kvm2 ovirt-ha-broker: self._backends[client].connect()
Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 456, in connect
Feb 17 01:25:41 kvm2 ovirt-ha-broker: self._dom_type)
Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 108, in get_domain_path
Feb 17 01:25:41 kvm2 ovirt-ha-broker: " in {1}".format(sd_uuid, parent))
Feb 17 01:25:41 kvm2 ovirt-ha-broker: BackendFailureException: path to
storage domain 7d376952-312b-4539-b809-a8fa740f7883 not found in
/rhev/data-center/mnt/glusterSD
Feb 17 01:25:41 kvm2 journal: vdsm vds ERROR failed to retrieve Hosted
Engine HA info#012Traceback (most recent call last):#012 File
"/usr/share/vdsm/API.py", line 1842, in _getHaInfo#012 stats =
instance.get_all_stats()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 103, in get_all_stats#012
self._configure_broker_conn(broker)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 180, in _configure_broker_conn#012 dom_type=dom_type)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 176, in set_storage_domain#012 .format(sd_type, options,
e))#012RequestError: Failed to set storage domain FilesystemBackend,
options {'dom_type': 'glusterfs', 'sd_uuid':
'7d376952-312b-4539-b809-a8fa740f7883'}: Request failed: <class
'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'>
Feb 17 01:25:41 kvm2 ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
closed
^C
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------060807010707030303030502
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 02/17/2016 04:57 PM, Luiz Claudio
Prazeres Goncalves wrote:<br>
</div>
<blockquote
cite="mid:CABUQ0hBjOXpMWN00EQsVAHVgLU4cznA2iYjPXRoh=HcXB1mbMg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Hi, I realised that gluster 3.7.8 was released for GA. So I
updated manually using "yum -y install glusterfs*" gluster
get's updated normally, but unfortunately things stopped to
work completely. ovirt_hosted_engine_ha was not able to
connect to my gluster storage domain (as you can see below).
I'm not using ovirt + gluster in hyperconverged way. I have 3
external gluster hosts. <br>
</div>
</div>
</blockquote>
<br>
<br>
What does the mount log on the hypervisor say? Could you attach
that? <br>
<br>
<blockquote
cite="mid:CABUQ0hBjOXpMWN00EQsVAHVgLU4cznA2iYjPXRoh=HcXB1mbMg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>As a workaround I've executed "<span
class="s1"><b>yum</b></span><span
class="s2"> downgrade glusterfs*" and after doing this
ovirt
started to work again...but as a collateral effect now I
can't migrate my vm's anymore. On the vdsm logs I can see
the following errors when trying to manually migrate a any
VM.... Anyone knows how to fix it? </span></div>
<div><span class="s2"><br>
</span></div>
<div><span class="s2">
<p class="p1"><span
class="s1">Thread-66055::DEBUG::2016-02-17
11:05:57,497::migration::453::virt.vm::(stop)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::stopping
migration downtime thread</span></p>
<p class="p1"><span
class="s1">Thread-66055::ERROR::2016-02-17
11:05:57,497::migration::208::virt.vm::(_recover)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::unable to
connect to server at '<a moz-do-not-send="true"
href="http://kvm2.brightsquid.com:49152">kvm2.brightsquid.com:49152</a>':
No route to host</span></p>
<p class="p1"><span
class="s1">Thread-66055::DEBUG::2016-02-17
11:05:57,497::stompreactor::389::jsonrpc.AsyncoreClient::(send)
Sending response</span></p>
<p class="p1"><span
class="s1">Thread-66056::DEBUG::2016-02-17
11:05:57,498::migration::450::virt.vm::(run)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::migration
downtime thread exiting</span></p>
<p class="p1"><span
class="s1">Thread-66055::DEBUG::2016-02-17
11:05:57,540::__init__::206::jsonrpc.Notification::(emit)
Sending event {"params": {"notify_time": 4327804740,
"05681896-a76a-4ae1-879e-8fe5d28634e1": {"status":
"Migration Source"}}, "jsonrpc": "2.0",
"method":
"|virt|VM_status|05681896-a76a-4ae1-879e-8fe5d28634e1"}</span></p>
<p class="p1"><span
class="s1">Thread-66055::ERROR::2016-02-17
11:05:57,541::migration::310::virt.vm::(run)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::</span><span
class="s2">Failed</span><span
class="s1"> to migrate</span></p>
<p class="p1"><span class="s1">Traceback (most
recent call
last):</span></p>
<p class="p1"><span class="s1"> File
"/usr/share/vdsm/virt/migration.py", line 294, in
run</span></p>
<p class="p1"><span class="s1">
self._startUnderlyingMigration(time.time())</span></p>
<p class="p1"><span class="s1"> File
"/usr/share/vdsm/virt/migration.py", line 364, in
_startUnderlyingMigration</span></p>
<p class="p1"><span class="s1">
self._perform_migration(duri, muri)</span></p>
<p class="p1"><span class="s1"> File
"/usr/share/vdsm/virt/migration.py", line 403, in
_perform_migration</span></p>
<p class="p1"><span class="s1">
self._vm._dom.migrateToURI3(duri, params, flags)</span></p>
<p class="p1"><span class="s1"> File
"/usr/share/vdsm/virt/virdomain.py", line 68, in
f</span></p>
<p class="p1"><span class="s1"> ret =
attr(*args,
**kwargs)</span></p>
<p class="p1"><span class="s1"> File
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 124, in wrapper</span></p>
<p class="p1"><span class="s1"> ret =
f(*args, **kwargs)</span></p>
<p class="p1"><span class="s1"> File
"/usr/lib64/python2.7/site-packages/libvirt.py", line
1836, in migrateToURI3</span></p>
<p class="p1"><span class="s1"> if ret ==
-1: raise
libvirtError ('virDomainMigrateToURI3() failed',
dom=self)</span></p>
<p class="p1"><span class="s1">libvirtError:
unable to
connect to server at '<a moz-do-not-send="true"
href="http://kvm2.brightsquid.com:49152">kvm2.brightsquid.com:49152</a>':
No route to host</span></p>
<p class="p1"><span
class="s1">Thread-68::DEBUG::2016-02-17
11:05:57,652::fileSD::173::Storage.Misc.excCmd::(getReadDelay)
/usr/bin/taskset --cpu-list 0-39 /usr/bin/dd
if=/rhev/data-center/mnt/gluster2.brightsquid.com:_home_export_iso/61827b7b-e255-44f5-a791-482a144be29f/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd
None)</span></p>
<p class="p1"><span
class="s1">Thread-68::DEBUG::2016-02-17
11:05:57,662::fileSD::173::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: <err> = '0+1 records in\n0+1 records
out\n341 bytes (341 B) copied, 0.000713967 s, 478
kB/s\n'; <rc> = 0</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
</span></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Issue to connect to the gluster storage.</div>
<div>*****************************************************************************</div>
<div>Feb 17 01:25:40 kvm2 ovirt-ha-agent:
<a class="moz-txt-link-freetext"
href="INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting">INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting</a>
storage server</div>
<div>Feb 17 01:25:40 kvm2 ovirt-ha-agent:
<a class="moz-txt-link-freetext"
href="INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting">INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting</a>
storage server</div>
<div>Feb 17 01:25:40 kvm2 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error:
'Connection to storage server failed' - trying to restart
agent</div>
<div>Feb 17 01:25:40 kvm2 ovirt-ha-agent:
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error:
'Connection to storage server failed' - trying to restart
agent</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker:
<a class="moz-txt-link-freetext"
href="INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection">INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection</a>
established</div>
<div>Feb 17 01:25:41 kvm2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR
Error handling request, data: 'set-storage-domain
FilesystemBackend dom_type=glusterfs
sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'#012Traceback
(most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 166, in handle#012 data)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 299, in _dispatch#012 .set_storage_domain(client,
sd_type, **options)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 66, in set_storage_domain#012
self._backends[client].connect()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 456, in connect#012 self._dom_type)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 108, in get_domain_path#012 " in {1}".format(sd_uuid,
parent))#012BackendFailureException: path to storage domain
7d376952-312b-4539-b809-a8fa740f7883 not found in
/rhev/data-center/mnt/glusterSD</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker:
ERROR:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Error
handling request, data: 'set-storage-domain FilesystemBackend
dom_type=glusterfs
sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: Traceback (most
recent call last):</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 166, in handle</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: data)</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 299, in _dispatch</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker:
.set_storage_domain(client, sd_type, **options)</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 66, in set_storage_domain</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker:
self._backends[client].connect()</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 456, in connect</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: self._dom_type)</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 108, in get_domain_path</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker: " in
{1}".format(sd_uuid, parent))</div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker:
BackendFailureException: path to storage domain
7d376952-312b-4539-b809-a8fa740f7883 not found in
/rhev/data-center/mnt/glusterSD</div>
<div>Feb 17 01:25:41 kvm2 journal: vdsm vds ERROR failed to
retrieve Hosted Engine HA info#012Traceback (most recent call
last):#012 File "/usr/share/vdsm/API.py", line 1842, in
_getHaInfo#012 stats = instance.get_all_stats()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 103, in get_all_stats#012
self._configure_broker_conn(broker)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 180, in _configure_broker_conn#012
dom_type=dom_type)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 176, in set_storage_domain#012 .format(sd_type,
options, e))#012RequestError: Failed to set storage domain
FilesystemBackend, options {'dom_type': 'glusterfs',
'sd_uuid': '7d376952-312b-4539-b809-a8fa740f7883'}: Request
failed: <class
'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'></div>
<div>Feb 17 01:25:41 kvm2 ovirt-ha-broker:
<a class="moz-txt-link-freetext"
href="INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection">INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection</a>
closed</div>
<div>^C</div>
<div><br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------060807010707030303030502--