12:19 a.m.
--------------090603010402080805030904
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Engine: oVirt Engine Version: 3.5.2-1.el7.centos
Nodes: oVirt Node - 3.5 - 0.999.201504280931.el7.centos
Remote storage: Dell Compellent SC8000
Storage setup: 2 nics connected to the Compellent. Several domains
backed by LUNs. Several VM disk using direct LUN.
Networking: Dell 10 Gb/s switches
I've been struggling with oVirt completely falling apart due to storage
related issues. By falling apart I mean most to all of the nodes
suddenly losing contact with the storage domains. This results in an
endless loop of the VMs on the failed nodes trying to be migrated and
remigrated as the nodes flap between response and unresponsive. During
these times, engine.log looks like this.
2015-05-19 03:09:42,443 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-50) domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
blade6c1.ism.ld
2015-05-19 03:09:42,560 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) domain
0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 in problem. vds:
blade2c1.ism.ld
2015-05-19 03:09:45,497 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-24) domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 in problem. vds:
blade3c2.ism.ld
2015-05-19 03:09:51,713 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-46) domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
blade4c2.ism.ld
2015-05-19 03:09:57,647 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-13) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
vds: blade6c1.ism.ld
2015-05-19 03:09:57,782 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-6) domain
26929b89-d1ca-4718-90d6-b3a6da585451:generic_data_1 in problem. vds:
blade2c1.ism.ld
2015-05-19 03:09:57,783 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-6) Domain
0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 recovered from problem.
vds: blade2c1.ism.ld
2015-05-19 03:10:00,639 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-31) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
vds: blade4c1.ism.ld
2015-05-19 03:10:00,703 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-17) domain
64101f40-0f10-471d-9f5f-44591f9e087d:logging_1 in problem. vds:
blade1c1.ism.ld
2015-05-19 03:10:00,712 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-4) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
vds: blade3c2.ism.ld
2015-05-19 03:10:06,931 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
vds: blade4c2.ism.ld
2015-05-19 03:10:06,931 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 has recovered from
problem. No active host in the DC is reporting it as problematic, so
clearing the domain recovery timer.
2015-05-19 03:10:06,932 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 recovered from problem.
vds: blade4c2.ism.ld
2015-05-19 03:10:06,933 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 has recovered from
problem. No active host in the DC is reporting it as problematic, so
clearing the domain recovery timer.
2015-05-19 03:10:09,929 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-16) domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
blade3c1.ism.ld
My troubleshooting steps so far:
1. Tailing engine.log for "in problem" and "recovered from problem"
2. Shutting down all the VMs.
3. Shutting down all but one node.
4. Bringing up one node at a time to see what the log reports.
When only one node is active everything is fine. When a second node
comes up, I begin to see the log output as shown above. I've been
struggling with this for over a month. I'm sure others have used oVirt
with a Compellent and encountered (and worked around) similar problems.
I'm looking for some help in figuring out if it's oVirt or something
that I'm doing wrong.
We're close to giving up on oVirt completely because of this.
P.S.
I've tested via bare metal and Proxmox with the Compellent. Not at the
same scale but it seems to work fine there.
--
This email was Virus checked by UTM 9. For issues please contact the Windows Systems
Admin.
--------------090603010402080805030904
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Engine:
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
oVirt Engine Version: 3.5.2-1.el7.centos<br>
Nodes: oVirt Node - 3.5 - 0.999.201504280931.el7.centos<br>
Remote storage: Dell Compellent SC8000<br>
Storage setup: 2 nics connected to the Compellent. Several domains
backed by LUNs. Several VM disk using direct LUN.<br>
Networking: Dell 10 Gb/s switches<br>
<br>
I've been struggling with oVirt completely falling apart due to
storage related issues. By falling apart I mean most to all of the
nodes suddenly losing contact with the storage domains. This results
in an endless loop of the VMs on the failed nodes trying to be
migrated and remigrated as the nodes flap between response and
unresponsive. During these times, engine.log looks like this.<br>
<br>
2015-05-19 03:09:42,443 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-50) domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
blade6c1.ism.ld<br>
2015-05-19 03:09:42,560 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) domain
0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 in problem. vds:
blade2c1.ism.ld<br>
2015-05-19 03:09:45,497 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-24) domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 in problem. vds:
blade3c2.ism.ld<br>
2015-05-19 03:09:51,713 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-46) domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
blade4c2.ism.ld<br>
2015-05-19 03:09:57,647 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-13) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from
problem. vds: blade6c1.ism.ld<br>
2015-05-19 03:09:57,782 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-6) domain
26929b89-d1ca-4718-90d6-b3a6da585451:generic_data_1 in problem. vds:
blade2c1.ism.ld<br>
2015-05-19 03:09:57,783 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-6) Domain
0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 recovered from
problem. vds: blade2c1.ism.ld<br>
2015-05-19 03:10:00,639 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-31) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from
problem. vds: blade4c1.ism.ld<br>
2015-05-19 03:10:00,703 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-17) domain
64101f40-0f10-471d-9f5f-44591f9e087d:logging_1 in problem. vds:
blade1c1.ism.ld<br>
2015-05-19 03:10:00,712 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-4) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from
problem. vds: blade3c2.ism.ld<br>
2015-05-19 03:10:06,931 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from
problem. vds: blade4c2.ism.ld<br>
2015-05-19 03:10:06,931 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 has recovered from
problem. No active host in the DC is reporting it as problematic, so
clearing the domain recovery timer.<br>
2015-05-19 03:10:06,932 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 recovered from
problem. vds: blade4c2.ism.ld<br>
2015-05-19 03:10:06,933 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 has recovered from
problem. No active host in the DC is reporting it as problematic, so
clearing the domain recovery timer.<br>
2015-05-19 03:10:09,929 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-16) domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
blade3c1.ism.ld<br>
<br>
<br>
My troubleshooting steps so far:<br>
<ol>
<li>Tailing engine.log for "in problem" and "recovered from
problem"</li>
<li>Shutting down all the VMs.</li>
<li>Shutting down all but one node.</li>
<li>Bringing up one node at a time to see what the log reports.</li>
</ol>
<p>When only one node is active everything is fine. When a second
node comes up, I begin to see the log output as shown above. I've
been struggling with this for over a month. I'm sure others have
used oVirt with a Compellent and encountered (and worked around)
similar problems. I'm looking for some help in figuring out if
it's oVirt or something that I'm doing wrong.<br>
</p>
<p>We're close to giving up on oVirt completely because of this.<br>
</p>
<p>P.S.<br>
</p>
<p>I've tested via bare metal and Proxmox with the Compellent. Not
at the same scale but it seems to work fine there.<br>
</p>
</body>
</html>
<pre>
--
This email was Virus checked by UTM 9. For issues please contact the Windows Systems
Admin.</pre>
--------------090603010402080805030904--
1:31 a.m.
Hello Chris,
On 05/19/2015 06:19 PM, Chris Jones - BookIt.com Systems Administrator
wrote:
Engine: oVirt Engine Version: 3.5.2-1.el7.centos
Nodes: oVirt Node - 3.5 - 0.999.201504280931.el7.centos
Remote storage: Dell Compellent SC8000
Storage setup: 2 nics connected to the Compellent. Several domains
backed by LUNs. Several VM disk using direct LUN.
Networking: Dell 10 Gb/s switches
I've been struggling with oVirt completely falling apart due to storage
related issues. By falling apart I mean most to all of the nodes
suddenly losing contact with the storage domains. This results in an
endless loop of the VMs on the failed nodes trying to be migrated and
remigrated as the nodes flap between response and unresponsive. During
these times, engine.log looks like this.
2015-05-19 03:09:42,443 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-50) domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
blade6c1.ism.ld
2015-05-19 03:09:42,560 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) domain
0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 in problem. vds:
blade2c1.ism.ld
2015-05-19 03:09:45,497 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-24) domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 in problem. vds:
blade3c2.ism.ld
2015-05-19 03:09:51,713 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-46) domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
blade4c2.ism.ld
2015-05-19 03:09:57,647 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-13) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
vds: blade6c1.ism.ld
2015-05-19 03:09:57,782 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-6) domain
26929b89-d1ca-4718-90d6-b3a6da585451:generic_data_1 in problem. vds:
blade2c1.ism.ld
2015-05-19 03:09:57,783 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-6) Domain
0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 recovered from problem.
vds: blade2c1.ism.ld
2015-05-19 03:10:00,639 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-31) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
vds: blade4c1.ism.ld
2015-05-19 03:10:00,703 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-17) domain
64101f40-0f10-471d-9f5f-44591f9e087d:logging_1 in problem. vds:
blade1c1.ism.ld
2015-05-19 03:10:00,712 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-4) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
vds: blade3c2.ism.ld
2015-05-19 03:10:06,931 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
vds: blade4c2.ism.ld
2015-05-19 03:10:06,931 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 has recovered from
problem. No active host in the DC is reporting it as problematic, so
clearing the domain recovery timer.
2015-05-19 03:10:06,932 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 recovered from problem.
vds: blade4c2.ism.ld
2015-05-19 03:10:06,933 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-48) Domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 has recovered from
problem. No active host in the DC is reporting it as problematic, so
clearing the domain recovery timer.
2015-05-19 03:10:09,929 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-16) domain
b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
blade3c1.ism.ld
My troubleshooting steps so far:
1. Tailing engine.log for "in problem" and "recovered from problem"
2. Shutting down all the VMs.
3. Shutting down all but one node.
4. Bringing up one node at a time to see what the log reports.
vdsm.log in the node side, will help here too.
When only one node is active everything is fine. When a second node
comes up, I begin to see the log output as shown above. I've been
struggling with this for over a month. I'm sure others have used oVirt
with a Compellent and encountered (and worked around) similar problems.
I'm looking for some help in figuring out if it's oVirt or something
that I'm doing wrong.
We're close to giving up on oVirt completely because of this.
P.S.
I've tested via bare metal and Proxmox with the Compellent. Not at the
same scale but it seems to work fine there.
Do you mind to share your vdsm.log from the oVirt Node machine?
To go to console in oVirt Node, press F2 in TUI.
Files: /var/log/vdsm/vdsm*.log
# rpm -qa | grep -i vdsm
might help too.
Thanks!
--
Cheers
Douglas
11:51 a.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
----- Original Message -----
Hello Chris,
On 05/19/2015 06:19 PM, Chris Jones - BookIt.com Systems Administrator
wrote:
> Engine: oVirt Engine Version: 3.5.2-1.el7.centos
> Nodes: oVirt Node - 3.5 - 0.999.201504280931.el7.centos
> Remote storage: Dell Compellent SC8000
> Storage setup: 2 nics connected to the Compellent. Several domains
> backed by LUNs. Several VM disk using direct LUN.
> Networking: Dell 10 Gb/s switches
>
> I've been struggling with oVirt completely falling apart due to storage
> related issues. By falling apart I mean most to all of the nodes
> suddenly losing contact with the storage domains. This results in an
> endless loop of the VMs on the failed nodes trying to be migrated and
> remigrated as the nodes flap between response and unresponsive. During
> these times, engine.log looks like this.
>
> 2015-05-19 03:09:42,443 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-50) domain
> c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
> blade6c1.ism.ld
> 2015-05-19 03:09:42,560 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) domain
> 0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 in problem. vds:
> blade2c1.ism.ld
> 2015-05-19 03:09:45,497 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-24) domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 in problem. vds:
> blade3c2.ism.ld
> 2015-05-19 03:09:51,713 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-46) domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
> blade4c2.ism.ld
> 2015-05-19 03:09:57,647 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-13) Domain
> c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
> vds: blade6c1.ism.ld
> 2015-05-19 03:09:57,782 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-6) domain
> 26929b89-d1ca-4718-90d6-b3a6da585451:generic_data_1 in problem. vds:
> blade2c1.ism.ld
> 2015-05-19 03:09:57,783 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-6) Domain
> 0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 recovered from problem.
> vds: blade2c1.ism.ld
> 2015-05-19 03:10:00,639 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-31) Domain
> c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
> vds: blade4c1.ism.ld
> 2015-05-19 03:10:00,703 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-17) domain
> 64101f40-0f10-471d-9f5f-44591f9e087d:logging_1 in problem. vds:
> blade1c1.ism.ld
> 2015-05-19 03:10:00,712 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-4) Domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
> vds: blade3c2.ism.ld
> 2015-05-19 03:10:06,931 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
> vds: blade4c2.ism.ld
> 2015-05-19 03:10:06,931 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 has recovered from
> problem. No active host in the DC is reporting it as problematic, so
> clearing the domain recovery timer.
> 2015-05-19 03:10:06,932 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 recovered from problem.
> vds: blade4c2.ism.ld
> 2015-05-19 03:10:06,933 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 has recovered from
> problem. No active host in the DC is reporting it as problematic, so
> clearing the domain recovery timer.
> 2015-05-19 03:10:09,929 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-16) domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
> blade3c1.ism.ld
>
>
> My troubleshooting steps so far:
>
> 1. Tailing engine.log for "in problem" and "recovered from
problem"
> 2. Shutting down all the VMs.
> 3. Shutting down all but one node.
> 4. Bringing up one node at a time to see what the log reports.
vdsm.log in the node side, will help here too.
> When only one node is active everything is fine. When a second node
> comes up, I begin to see the log output as shown above. I've been
> struggling with this for over a month. I'm sure others have used oVirt
> with a Compellent and encountered (and worked around) similar problems.
> I'm looking for some help in figuring out if it's oVirt or something
> that I'm doing wrong.
>
> We're close to giving up on oVirt completely because of this.
>
> P.S.
>
> I've tested via bare metal and Proxmox with the Compellent. Not at the
> same scale but it seems to work fine there.
Do you mind to share your vdsm.log from the oVirt Node machine?
To go to console in oVirt Node, press F2 in TUI.
Files: /var/log/vdsm/vdsm*.log
# rpm -qa | grep -i vdsm
might help too.
Hey Chris,
please open a bug [1] for this, then we can track it and we can help to
identify the issue.
I'm not seeing anything suspicious from the logs above. But adding the
logs Douglas mentions above to the new bug will help us to get more insight.
Greetings
fabian
---
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt&component=ovi...
11:49 p.m.
> vdsm.log in the node side, will help here too.
https://www.dropbox.com/s/zvnttmylmrd0hyx/vdsm.log.gz?dl=0. This log
contains only the messages at and after when a host was become
unresponsive due to storage issues.
> # rpm -qa | grep -i vdsm
> might help too.
vdsm-cli-4.16.14-0.el7.noarch
vdsm-reg-4.16.14-0.el7.noarch
ovirt-node-plugin-vdsm-0.2.2-5.el7.noarch
vdsm-python-zombiereaper-4.16.14-0.el7.noarch
vdsm-xmlrpc-4.16.14-0.el7.noarch
vdsm-yajsonrpc-4.16.14-0.el7.noarch
vdsm-4.16.14-0.el7.x86_64
vdsm-gluster-4.16.14-0.el7.noarch
vdsm-hook-ethtool-options-4.16.14-0.el7.noarch
vdsm-python-4.16.14-0.el7.noarch
vdsm-jsonrpc-4.16.14-0.el7.noarch
Hey Chris,
please open a bug [1] for this, then we can track it and we can help to
identify the issue.
I will do so.
1:10 a.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
----- Original Message -----
From: "Chris Jones - BookIt.com Systems Administrator"
<chris.jones(a)bookit.com>
To: users(a)ovirt.org
Sent: Thursday, May 21, 2015 12:49:50 AM
Subject: Re: [ovirt-users] oVirt Instability with Dell Compellent via iSCSI/Multipath
>> vdsm.log in the node side, will help here too.
https://www.dropbox.com/s/zvnttmylmrd0hyx/vdsm.log.gz?dl=0. This log
contains only the messages at and after when a host was become
unresponsive due to storage issues.
According to the log, you have a real issue accessing storage from the host:
[nsoffer@thin untitled (master)]$ repostat vdsm.log
domain: 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2
delay avg: 0.000856 min: 0.000000 max: 0.001168
last check avg: 11.510000 min: 0.300000 max: 64.100000
domain: 64101f40-0f10-471d-9f5f-44591f9e087d
delay avg: 0.008358 min: 0.000000 max: 0.040269
last check avg: 11.863333 min: 0.300000 max: 63.400000
domain: 31e97cc8-6a10-4a45-8f25-95eba88b4dc0
delay avg: 0.007793 min: 0.000819 max: 0.041316
last check avg: 11.466667 min: 0.000000 max: 70.200000
domain: 842edf83-22c6-46cd-acaa-a1f76d61e545
delay avg: 0.000493 min: 0.000374 max: 0.000698
last check avg: 4.860000 min: 0.200000 max: 9.900000
domain: b050c455-5ab1-4107-b055-bfcc811195fc
delay avg: 0.002080 min: 0.000000 max: 0.040142
last check avg: 11.830000 min: 0.000000 max: 63.700000
domain: c46adffc-614a-4fa2-9d2d-954f174f6a39
delay avg: 0.004798 min: 0.000000 max: 0.041006
last check avg: 18.423333 min: 1.400000 max: 102.900000
domain: 0b1d36e4-7992-43c7-8ac0-740f7c2cadb7
delay avg: 0.001002 min: 0.000000 max: 0.001199
last check avg: 11.560000 min: 0.300000 max: 61.700000
domain: 20153412-f77a-4944-b252-ff06a78a1d64
delay avg: 0.003748 min: 0.000000 max: 0.040903
last check avg: 12.180000 min: 0.000000 max: 67.200000
domain: 26929b89-d1ca-4718-90d6-b3a6da585451
delay avg: 0.000963 min: 0.000000 max: 0.001209
last check avg: 10.993333 min: 0.000000 max: 64.300000
domain: 0137183b-ea40-49b1-b617-256f47367280
delay avg: 0.000881 min: 0.000000 max: 0.001227
last check avg: 11.086667 min: 0.100000 max: 63.200000
Note the high last check maximum value (e.g. 102 seconds).
Vdsm has a monitor thread for each domain, doing a read from one of the storage
domain special disk every 10 seconds. When we see high last check value, it
means that the monitor thread is stuck reading from the disk.
This is an indicator that vms may have trouble accessing this storage domains,
and engine is handling this by making the host non-operational, or if all hosts
cannot access the domain, making the domain inactive.
One of the known issues that can be related, is bad multipath configuration. Some
storage server have bad builtin configuration embedded into multipath. In particular,
using "no_path_retry queue", or "no_path_retry 60". This setting means
that when
the SCSI layer fails, and multipath does not have any active path it will queue
io foerver (queue), or retry many times (e.g, 60) before failing the io request.
This can lead to stuck process, doing a read or write that never fails or takes
many minutes to fail. Vdsm is not designed to handle such delays - a stuck thread
may block other unrelated threads.
Vdsm includes special configuration for your storage vendor (COMPELNT), but maybe
it does not match the product (Compellent Vol).
See
https://github.com/oVirt/vdsm/blob/master/lib/vdsm/tool/configurators/mul...
device {
vendor "COMPELNT"
product "Compellent Vol"
no_path_retry fail
}
Another issue may be that the setting for COMPELNT/Compellent Vol are wrong;
the setting we ship is missing lot of settings that exists in the builtin
setting, and this may have bad effect. If your devices match this , I would
try this multipath configuration, instead of the one vdsm configures.
device {
vendor "COMPELNT"
product "Compellent Vol"
path_grouping_policy "multibus"
path_checker "tur"
features "0"
hardware_handler "0"
prio "const"
failback "immediate"
rr_weight "uniform"
no_path_retry fail
}
To verify that your devices match this, you can check the devices vendor and procut
strings in the output of "multipath -ll". I would like to see the output of this
command.
Another platform issue is bad default SCSI node.session.timeo.replacement_timeout
value, which is set to 120 seconds. This setting mean that the SCSI layer will
wait 120 seconds for io to complete on one path, before failing the io request.
So you may have one bad path, causing 120 second delay, while you could complete
the request using another path.
Multipath is trying to set this value to 5 seconds, but this value is reverting
to the default 120 seconds after a device has trouble. There is an open bug about
this which we hope to get fixed in the rhel/centos 7.2.
https://bugzilla.redhat.com/1139038
This issue together with "no_path_retry queue" is a very bad mix for ovirt.
You can fix this timeout by setting:
# /etc/iscsi/iscsid.conf
node.session.timeo.replacement_timeout = 5
And restarting iscsid service.
With these tweaks, the issue may be resolved.
I hope it helps.
Nir
>> # rpm -qa | grep -i vdsm
>> might help too.
vdsm-cli-4.16.14-0.el7.noarch
vdsm-reg-4.16.14-0.el7.noarch
ovirt-node-plugin-vdsm-0.2.2-5.el7.noarch
vdsm-python-zombiereaper-4.16.14-0.el7.noarch
vdsm-xmlrpc-4.16.14-0.el7.noarch
vdsm-yajsonrpc-4.16.14-0.el7.noarch
vdsm-4.16.14-0.el7.x86_64
vdsm-gluster-4.16.14-0.el7.noarch
vdsm-hook-ethtool-options-4.16.14-0.el7.noarch
vdsm-python-4.16.14-0.el7.noarch
vdsm-jsonrpc-4.16.14-0.el7.noarch
>
> Hey Chris,
>
> please open a bug [1] for this, then we can track it and we can help to
> identify the issue.
I will do so.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:31 a.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
On 05/20/2015 07:10 PM, Nir Soffer wrote:
----- Original Message -----
> From: "Chris Jones - BookIt.com Systems Administrator"
<chris.jones(a)bookit.com>
> To: users(a)ovirt.org
> Sent: Thursday, May 21, 2015 12:49:50 AM
> Subject: Re: [ovirt-users] oVirt Instability with Dell Compellent
via iSCSI/Multipath
>
>>> vdsm.log in the node side, will help here too.
>
> https://www.dropbox.com/s/zvnttmylmrd0hyx/vdsm.log.gz?dl=0. This log
> contains only the messages at and after when a host was become
> unresponsive due to storage issues.
According to the log, you have a real issue accessing storage from the host:
[nsoffer@thin untitled (master)]$ repostat vdsm.log
domain: 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2
delay avg: 0.000856 min: 0.000000 max: 0.001168
last check avg: 11.510000 min: 0.300000 max: 64.100000
domain: 64101f40-0f10-471d-9f5f-44591f9e087d
delay avg: 0.008358 min: 0.000000 max: 0.040269
last check avg: 11.863333 min: 0.300000 max: 63.400000
domain: 31e97cc8-6a10-4a45-8f25-95eba88b4dc0
delay avg: 0.007793 min: 0.000819 max: 0.041316
last check avg: 11.466667 min: 0.000000 max: 70.200000
domain: 842edf83-22c6-46cd-acaa-a1f76d61e545
delay avg: 0.000493 min: 0.000374 max: 0.000698
last check avg: 4.860000 min: 0.200000 max: 9.900000
domain: b050c455-5ab1-4107-b055-bfcc811195fc
delay avg: 0.002080 min: 0.000000 max: 0.040142
last check avg: 11.830000 min: 0.000000 max: 63.700000
domain: c46adffc-614a-4fa2-9d2d-954f174f6a39
delay avg: 0.004798 min: 0.000000 max: 0.041006
last check avg: 18.423333 min: 1.400000 max: 102.900000
domain: 0b1d36e4-7992-43c7-8ac0-740f7c2cadb7
delay avg: 0.001002 min: 0.000000 max: 0.001199
last check avg: 11.560000 min: 0.300000 max: 61.700000
domain: 20153412-f77a-4944-b252-ff06a78a1d64
delay avg: 0.003748 min: 0.000000 max: 0.040903
last check avg: 12.180000 min: 0.000000 max: 67.200000
domain: 26929b89-d1ca-4718-90d6-b3a6da585451
delay avg: 0.000963 min: 0.000000 max: 0.001209
last check avg: 10.993333 min: 0.000000 max: 64.300000
domain: 0137183b-ea40-49b1-b617-256f47367280
delay avg: 0.000881 min: 0.000000 max: 0.001227
last check avg: 11.086667 min: 0.100000 max: 63.200000
Note the high last check maximum value (e.g. 102 seconds).
Vdsm has a monitor thread for each domain, doing a read from one of the storage
domain special disk every 10 seconds. When we see high last check value, it
means that the monitor thread is stuck reading from the disk.
This is an indicator that vms may have trouble accessing this storage domains,
and engine is handling this by making the host non-operational, or if all hosts
cannot access the domain, making the domain inactive.
One of the known issues that can be related, is bad multipath configuration. Some
storage server have bad builtin configuration embedded into multipath. In particular,
using "no_path_retry queue", or "no_path_retry 60". This setting
means that when
the SCSI layer fails, and multipath does not have any active path it will queue
io foerver (queue), or retry many times (e.g, 60) before failing the io request.
This can lead to stuck process, doing a read or write that never fails or takes
many minutes to fail. Vdsm is not designed to handle such delays - a stuck thread
may block other unrelated threads.
Vdsm includes special configuration for your storage vendor (COMPELNT), but maybe
it does not match the product (Compellent Vol).
See
https://github.com/oVirt/vdsm/blob/master/lib/vdsm/tool/configurators/mul...
device {
vendor "COMPELNT"
product "Compellent Vol"
no_path_retry fail
}
Another issue may be that the setting for COMPELNT/Compellent Vol are wrong;
the setting we ship is missing lot of settings that exists in the builtin
setting, and this may have bad effect. If your devices match this , I would
try this multipath configuration, instead of the one vdsm configures.
device {
vendor "COMPELNT"
product "Compellent Vol"
path_grouping_policy "multibus"
path_checker "tur"
features "0"
hardware_handler "0"
prio "const"
failback "immediate"
rr_weight "uniform"
no_path_retry fail
}
To verify that your devices match this, you can check the devices vendor and procut
strings in the output of "multipath -ll". I would like to see the output of
this
command.
Another platform issue is bad default SCSI node.session.timeo.replacement_timeout
value, which is set to 120 seconds. This setting mean that the SCSI layer will
wait 120 seconds for io to complete on one path, before failing the io request.
So you may have one bad path, causing 120 second delay, while you could complete
the request using another path.
Multipath is trying to set this value to 5 seconds, but this value is reverting
to the default 120 seconds after a device has trouble. There is an open bug about
this which we hope to get fixed in the rhel/centos 7.2.
https://bugzilla.redhat.com/1139038
This issue together with "no_path_retry queue" is a very bad mix for ovirt.
You can fix this timeout by setting:
# /etc/iscsi/iscsid.conf
node.session.timeo.replacement_timeout = 5
And restarting iscsid service.
Chris, as you are using ovirt-node, after Nir suggestions please also
execute the below command too to save the settings changes across reboots:
# persist /etc/iscsi/iscsid.conf
With these tweaks, the issue may be resolved.
I hope it helps.
Nir
>
>>> # rpm -qa | grep -i vdsm
>>> might help too.
>
> vdsm-cli-4.16.14-0.el7.noarch
> vdsm-reg-4.16.14-0.el7.noarch
> ovirt-node-plugin-vdsm-0.2.2-5.el7.noarch
> vdsm-python-zombiereaper-4.16.14-0.el7.noarch
> vdsm-xmlrpc-4.16.14-0.el7.noarch
> vdsm-yajsonrpc-4.16.14-0.el7.noarch
> vdsm-4.16.14-0.el7.x86_64
> vdsm-gluster-4.16.14-0.el7.noarch
> vdsm-hook-ethtool-options-4.16.14-0.el7.noarch
> vdsm-python-4.16.14-0.el7.noarch
> vdsm-jsonrpc-4.16.14-0.el7.noarch
>
>>
>> Hey Chris,
>>
>> please open a bug [1] for this, then we can track it and we can help to
>> identify the issue.
>
> I will do so.
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Cheers
Douglas
2:49 a.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
Chris, as you are using ovirt-node, after Nir suggestions please
also
execute the below command too to save the settings changes across reboots:
# persist /etc/iscsi/iscsid.conf
Thanks. I will do so, but first I have to resolve not being able to
update multipath.conf as described in my previous email.
2:47 a.m.
Another issue may be that the setting for COMPELNT/Compellent Vol are
wrong;
the setting we ship is missing lot of settings that exists in the builtin
setting, and this may have bad effect. If your devices match this , I would
try this multipath configuration, instead of the one vdsm configures.
device {
vendor "COMPELNT"
product "Compellent Vol"
path_grouping_policy "multibus"
path_checker "tur"
features "0"
hardware_handler "0"
prio "const"
failback "immediate"
rr_weight "uniform"
no_path_retry fail
}
I wish I could. We're using the CentOS 7 ovirt-node-iso. The
multipath.conf is less than ideal but when I tried updating it, oVirt
instantly overwrites it. To be clear, yes I know changes do not survive
reboots and yes I know about persist, but it changes it while running.
Live! Persist won't help there.
I also tried building a CentOS 7 "thick client" where I set up CentOS 7
first, added the oVirt repo, then let the engine provision it. Same
problem with multipath.conf being overwritten with the default oVirt setup.
So I tried to be slick about it. I made the multipath.conf immutable.
That prevented the engine from being able to activate the node. It would
fail on a vds command that gets the nodes capabilities and part of what
it does is reads then overwrites multipath.conf.
How do I safely update multipath.conf?
To verify that your devices match this, you can check the devices vendor and procut
strings in the output of "multipath -ll". I would like to see the output of
this
command.
multipath -ll (default setup) can be seen here.
http://paste.linux-help.org/view/430c7538
Another platform issue is bad default SCSI
node.session.timeo.replacement_timeout
value, which is set to 120 seconds. This setting mean that the SCSI layer will
wait 120 seconds for io to complete on one path, before failing the io request.
So you may have one bad path, causing 120 second delay, while you could complete
the request using another path.
Multipath is trying to set this value to 5 seconds, but this value is reverting
to the default 120 seconds after a device has trouble. There is an open bug about
this which we hope to get fixed in the rhel/centos 7.2.
https://bugzilla.redhat.com/1139038
This issue together with "no_path_retry queue" is a very bad mix for ovirt.
You can fix this timeout by setting:
# /etc/iscsi/iscsid.conf
node.session.timeo.replacement_timeout = 5
I'll see if that's possible with persist. Will this change survive node
upgrades?
Thanks for the reply and the suggestions.
11:19 a.m.
This is a multi-part message in MIME format.
------------MIME-1975911516-1580342552-delim
Content-Type: text/plain;
charset="windows-1252"
Content-Transfer-Encoding: quoted-printable
On 05/21/2015 02=3A47 AM=2C Chris Jones - BookIt=2Ecom Systems Administrato=
r
wrote=3A
=3E=3E Another issue may be that the setting for COMPELNT/Compellent Vol ar=
e
=3E=3E wrong=3B
=3E=3E the setting we ship is missing lot of settings that exists in the
=3E=3E builtin
=3E=3E setting=2C and this may have bad effect=2E If your devices match thi=
s =2C I
=3E=3E would
=3E=3E try this multipath configuration=2C instead of the one vdsm configur=
es=2E
=3E=3E
=3E=3E device =7B
=3E=3E vendor =22COMPELNT=22
=3E=3E product =22Compellent Vol=22
=3E=3E path=5Fgrouping=5Fpolicy =22multibus=22
=3E=3E path=5Fchecker =22tur=22
=3E=3E features =220=22
=3E=3E hardware=5Fhandler =220=22
=3E=3E prio =22const=22
=3E=3E failback =22immediate=22
=3E=3E rr=5Fweight =22uniform=22
=3E=3E no=5Fpath=5Fretry fail
=3E=3E =7D
=3E
=3E I wish I could=2E We=27re using the CentOS 7 ovirt-node-iso=2E The
=3E multipath=2Econf is less than ideal but when I tried updating it=2C oVi=
rt
=3E instantly overwrites it=2E To be clear=2C yes I know changes do not
=3E survive reboots and yes I know about persist=2C but it changes it while=
=3E running=2E Live! Persist won=27t help there=2E
=3E
=3E I also tried building a CentOS 7 =22thick client=22 where I set up Cent=
OS
=3E 7 first=2C added the oVirt repo=2C then let the engine provision it=2E=
Same
=3E problem with multipath=2Econf being overwritten with the default oVirt=
=3E setup=2E
=3E
=3E So I tried to be slick about it=2E I made the multipath=2Econf immutabl=
e=2E
=3E That prevented the engine from being able to activate the node=2E It
=3E would fail on a vds command that gets the nodes capabilities and part=
=3E of what it does is reads then overwrites multipath=2Econf=2E
=3E
=3E How do I safely update multipath=2Econf=3F
=3E
Somehow the multipath=2Econf that oVirt generates forces my HDD RAID
controller disks to change from /dev/sdb* and /dev/sdc*=2E So I had to
blacklist these=2E
I was able to persist it by adding =22=23 RHEV PRIVATE=22 right below the=
=22=23
RHEV REVISION 1=2E1=22
Hope this helps
Met vriendelijke groet=2C With kind regards=2C
Jorick Astrego
Netbulae Virtualization Experts=20
----------------
=09Tel=3A 053 20 30 270 =09info=40netbulae=2Eeu =09Staalsteden 4-3A =09KvK=
08198180
=09Fax=3A 053 20 30 271 =09www=2Enetbulae=2Eeu =097547 TA Enschede =09BTW=
NL821234584B01
----------------
------------MIME-1975911516-1580342552-delim
Content-Type: text/html;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
=3Chtml=3E
=3Cbody=3E
<br>
<br>
On 05/21/2015 02:47 AM, Chris Jones - BookIt.com Systems Administrator =
<br>
wrote: <br>
<font color=3D"#000000">>> Another issue may be that the
setting for =
COMPELNT/Compellent Vol are </font><br>
<font color=3D"#000000">>>
wrong; </font><br>
<font color=3D"#000000">>> the setting we ship is missing
lot of sett=
ings that exists in the </font><br>
<font color=3D"#000000">>>
builtin </font><br>
<font color=3D"#000000">>> setting, and this may have bad
effect. If =
your devices match this , I </font><br>
<font color=3D"#000000">>>
would </font><br>
<font color=3D"#000000">>> try this multipath configuration,
instead =
of the one vdsm configures. </font><br>
<font color=3D"#000000">>> </font><br>
<font
color=3D"#000000">>>
device { </fon=
t><br>
<font
color=3D"#000000">>> &=
nbsp; vendor "COMPELNT" </font><br>
<font
color=3D"#000000">>> &=
nbsp; product "Compellent Vol" </font><br>
<font
color=3D"#000000">>> &=
nbsp; path_grouping_policy "multibus" </font><br>
<font
color=3D"#000000">>> &=
nbsp; path_checker "tur" </font><br>
<font
color=3D"#000000">>> &=
nbsp; features "0" </font><br>
<font
color=3D"#000000">>> &=
nbsp; hardware_handler "0" </font><br>
<font
color=3D"#000000">>> &=
nbsp; prio "const" </font><br>
<font
color=3D"#000000">>> &=
nbsp; failback "immediate" </font><br>
<font
color=3D"#000000">>> &=
nbsp; rr_weight "uniform" </font><br>
<font
color=3D"#000000">>> &=
nbsp; no_path_retry fail </font><br>
<font
color=3D"#000000">>>
} </font><br>
<font color=3D"#000000">> </font><br>
<font color=3D"#000000">> I wish I could. We're using the
CentOS 7 ovirt=
-node-iso. The </font><br>
<font color=3D"#000000">> multipath.conf is less than ideal but
when I t=
ried updating it, oVirt </font><br>
<font color=3D"#000000">> instantly overwrites it. To be clear, yes
I kn=
ow changes do not </font><br>
<font color=3D"#000000">> survive reboots and yes I know about
persist, =
but it changes it while </font><br>
<font color=3D"#000000">> running. Live! Persist won't help
there. <=
/font><br>
<font color=3D"#000000">> </font><br>
<font color=3D"#000000">> I also tried building a CentOS 7
"thick c=
lient" where I set up CentOS </font><br>
<font color=3D"#000000">> 7 first, added the oVirt repo, then let
the en=
gine provision it. Same </font><br>
<font color=3D"#000000">> problem with multipath.conf being
overwritten =
with the default oVirt </font><br>
<font color=3D"#000000">> setup. </font><br>
<font color=3D"#000000">> </font><br>
<font color=3D"#000000">> So I tried to be slick about it. I made
the mu=
ltipath.conf immutable. </font><br>
<font color=3D"#000000">> That prevented the engine from being able
to a=
ctivate the node. It </font><br>
<font color=3D"#000000">> would fail on a vds command that gets the
node=
s capabilities and part </font><br>
<font color=3D"#000000">> of what it does is reads then overwrites
multi=
path.conf. </font><br>
<font color=3D"#000000">> </font><br>
<font color=3D"#000000">> How do I safely update
multipath.conf? </f=
ont><br>
<font color=3D"#000000">> </font><br>
<br>
Somehow the multipath.conf that oVirt generates forces my HDD RAID <br>
controller disks to change from /dev/sdb* and /dev/sdc*. So I had to <b=
r>
blacklist these. <br>
<br>
I was able to persist it by adding "# RHEV PRIVATE" right below t=
he "# <br>
RHEV REVISION 1.1" <br>
<br>
Hope this helps <br>
<br>
<br>
<br>
=
=3CBR /=3E
=3CBR /=3E
=3Cb style=3D=22color=3A=23604c78=22=3E=3C/b=3E=3Cbr=3E=3Cspan style=3D=22c=
olor=3A=23604c78=3B=22=3E=3Cfont color=3D=22000000=22=3E=3Cspan style=3D=22=
mso-fareast-language=3Aen-gb=3B=22 lang=3D=22NL=22=3EMet vriendelijke groet=
=2C With kind regards=2C=3Cbr=3E=3Cbr=3E=3C/span=3EJorick Astrego=3C/font=
=3E=3C/span=3E=3Cb style=3D=22color=3A=23604c78=22=3E=3Cbr=3E=3Cbr=3ENetbul=
ae Virtualization Experts =3C/b=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3B=
border-top=3A1px solid =23ccc=3B=22=3E=3Ctable style=3D=22width=3A 522px=22=
=3E=3Ctbody=3E=3Ctr=3E=3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=
=22=3ETel=3A 053 20 30 270=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bf=
ont-size=3A 10px=22=3Einfo=40netbulae=2Eeu=3C/td=3E =3Ctd style=3D=22wid=
th=3A 130px=3Bfont-size=3A 10px=22=3EStaalsteden 4-3A=3C/td=3E =3Ctd sty=
le=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EKvK 08198180=3C/td=3E=3C/tr=
=3E=3Ctr=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EFax=
=3A 053 20 30 271=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=
=3A 10px=22=3Ewww=2Enetbulae=2Eeu=3C/td=3E =3Ctd style=3D=22width=3A 130=
px=3Bfont-size=3A 10px=22=3E7547 TA Enschede=3C/td=3E =3Ctd style=3D=22w=
idth=3A 130px=3Bfont-size=3A 10px=22=3EBTW NL821234584B01=3C/td=3E=3C/tr=3E=
=3C/tbody=3E=3C/table=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3Bborder-top=
=3A1px solid =23ccc=3B=22=3E=3CBR /=3E
=3C/body=3E
=3C/html=3E
------------MIME-1975911516-1580342552-delim--
11:25 a.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
On 21.05.2015 02:48, Chris Jones - BookIt.com Systems Administrator wrote:
> Another issue may be that the setting for COMPELNT/Compellent Vol
are wrong;
> the setting we ship is missing lot of settings that exists in the builtin
> setting, and this may have bad effect. If your devices match this , I would
> try this multipath configuration, instead of the one vdsm configures.
>
> device {
> vendor "COMPELNT"
> product "Compellent Vol"
> path_grouping_policy "multibus"
> path_checker "tur"
> features "0"
> hardware_handler "0"
> prio "const"
> failback "immediate"
> rr_weight "uniform"
> no_path_retry fail
> }
I wish I could. We're using the CentOS 7 ovirt-node-iso. The
multipath.conf is less than ideal
I have this issue also. I think about opening a
BZ ;)
but when I tried updating it, oVirt
instantly overwrites it. To be clear, yes I know changes do not
survive
reboots and yes I know about persist, but it changes it while running.
Live! Persist won't help there.
I also tried building a CentOS 7 "thick client" where I set up CentOS 7
first, added the oVirt repo, then let the engine provision it. Same
problem with multipath.conf being overwritten with the default oVirt setup.
So I tried to be slick about it. I made the multipath.conf immutable.
That prevented the engine from being able to activate the node. It would
fail on a vds command that gets the nodes capabilities and part of what
it does is reads then overwrites multipath.conf.
How do I safely update multipath.conf?
In the second line of your multipath conf, add:
# RHEV PRIVATE
Then, host deploy will ignore it and never change it.
>
> To verify that your devices match this, you can check the devices vendor and procut
> strings in the output of "multipath -ll". I would like to see the output of
this
> command.
multipath -ll (default setup) can be seen here.
http://paste.linux-help.org/view/430c7538
> Another platform issue is bad default SCSI node.session.timeo.replacement_timeout
> value, which is set to 120 seconds. This setting mean that the SCSI layer will
> wait 120 seconds for io to complete on one path, before failing the io request.
> So you may have one bad path, causing 120 second delay, while you could complete
> the request using another path.
>
> Multipath is trying to set this value to 5 seconds, but this value is reverting
> to the default 120 seconds after a device has trouble. There is an open bug about
> this which we hope to get fixed in the rhel/centos 7.2.
> https://bugzilla.redhat.com/1139038
>
> This issue together with "no_path_retry queue" is a very bad mix for
ovirt.
>
> You can fix this timeout by setting:
>
> # /etc/iscsi/iscsid.conf
> node.session.timeo.replacement_timeout = 5
I'll see if that's possible with persist. Will this change survive node
upgrades?
Thanks for the reply and the suggestions.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
9:49 p.m.
I've applied the multipath.conf and iscsi.conf changes you recommended.
It seems to be running better. I was able to bring up all the hosts and
VMs without it falling apart.
I'm still seeing the domain "in problem" and "recovered" from
problem
warnings in engine.log, though. They were happening only when hosts were
activating and when I was mass launching many VMs. Is this normal?
2015-05-21 15:31:32,264 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-13) domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
blade6c2.ism.ld
2015-05-21 15:31:47,468 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-4) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
vds: blade6c2.ism.ld
Here's the vdsm log from a node the engine was warning about
https://www.dropbox.com/s/yaubaxax1w499f1/vdsm2.log.gz?dl=1. It's
trimmed to just before and after it happened.
What is that repo stat command from your previous email, Nir? "repostat
vdsm.log" I don't see it on the engine or the node. Is it used to parse
the log? Where can I find it?
Thanks again.
11:32 p.m.
On 05/21/2015 03:49 PM, Chris Jones - BookIt.com Systems Administrator
wrote:
I've applied the multipath.conf and iscsi.conf changes you
recommended.
It seems to be running better. I was able to bring up all the hosts and
VMs without it falling apart.
I take it back. This did not solve the issue. I tried batch starting the
VMs and half the nodes went down due to the same storage issues. VDSM
Logs again.
https://www.dropbox.com/s/12sudzhaily72nb/vdsm_failures.log.gz?dl=1
6:13 p.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
----- Original Message -----
On 05/21/2015 03:49 PM, Chris Jones - BookIt.com Systems
Administrator
wrote:
> I've applied the multipath.conf and iscsi.conf changes you recommended.
> It seems to be running better. I was able to bring up all the hosts and
> VMs without it falling apart.
I take it back. This did not solve the issue. I tried batch starting the
VMs and half the nodes went down due to the same storage issues. VDSM
Is there maybe some IO problem on the iSCSI target side?
IIUIC the problem is some timeout, which could indicate that the target
is overloaded.
But maybe I get something wrong ...
- fabian
Logs again.
https://www.dropbox.com/s/12sudzhaily72nb/vdsm_failures.log.gz?dl=1
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
7:55 p.m.
Is there maybe some IO problem on the iSCSI target side?
IIUIC the problem is some timeout, which could indicate that the target
is overloaded.
Maybe. I need to check with Dell. I did manage to get it to be a little
more stable with this config.
defaults {
polling_interval 10
path_selector "round-robin 0"
path_grouping_policy multibus
getuid_callout "/usr/lib/udev/scsi_id --whitelisted
--replace-whitespace --device=/dev/%n"
path_checker readsector0
rr_min_io_rq 100
max_fds 8192
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_names no
}
devices {
device {
vendor COMPELNT
product "Compellent Vol"
path_checker tur
no_path_retry fail
}
}
I referenced it from
http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_sol....
I modified it a bit since that is Red Hat 5 specific and there have been
some changes.
It's not crashing anymore but I'm still seeing storage warnings in
engine.log. I'm going to be enabling jumbo frames and talking with Dell
to figure out if it's something on the Compellent side. I'll update here
once I find something out.
Thanks again for all the help.
8:25 p.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
----- Original Message -----
From: "Chris Jones - BookIt.com Systems Administrator"
<chris.jones(a)bookit.com>
To: users(a)ovirt.org
Sent: Friday, May 22, 2015 8:55:37 PM
Subject: Re: [ovirt-users] oVirt Instability with Dell Compellent via iSCSI/Multipath
> Is there maybe some IO problem on the iSCSI target side?
> IIUIC the problem is some timeout, which could indicate that the target
> is overloaded.
Maybe. I need to check with Dell. I did manage to get it to be a little
more stable with this config.
defaults {
polling_interval 10
path_selector "round-robin 0"
path_grouping_policy multibus
getuid_callout "/usr/lib/udev/scsi_id --whitelisted
--replace-whitespace --device=/dev/%n"
path_checker readsector0
rr_min_io_rq 100
max_fds 8192
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_names no
You should keep the default without change, and add specific settings under
the device section.
}
devices {
device {
vendor COMPELNT
product "Compellent Vol"
path_checker tur
no_path_retry fail
This is mostly likely missing some settings. You are *not* getting the settings
from the "defaults" section above.
For example, since you did not specify here "failback immediate", failback for
this device defaults to whatever default multipath chose, not the value set
in "defaults" above.
}
}
I referenced it from
http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_sol....
I modified it a bit since that is Red Hat 5 specific and there have been
some changes.
It's not crashing anymore but I'm still seeing storage warnings in
engine.log. I'm going to be enabling jumbo frames and talking with Dell
to figure out if it's something on the Compellent side. I'll update here
once I find something out.
Lets continue this on bugzilla.
See also this patch:
https://gerrit.ovirt.org/41244
Nir
8:08 p.m.
8:18 p.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
----- Original Message -----
From: "Chris Jones - BookIt.com Systems Administrator"
<chris.jones(a)bookit.com>
To: users(a)ovirt.org
Sent: Friday, May 22, 2015 12:32:01 AM
Subject: Re: [ovirt-users] oVirt Instability with Dell Compellent via iSCSI/Multipath
On 05/21/2015 03:49 PM, Chris Jones - BookIt.com Systems Administrator
wrote:
> I've applied the multipath.conf and iscsi.conf changes you recommended.
> It seems to be running better. I was able to bring up all the hosts and
> VMs without it falling apart.
I take it back. This did not solve the issue. I tried batch starting the
VMs and half the nodes went down due to the same storage issues. VDSM
Logs again.
https://www.dropbox.com/s/12sudzhaily72nb/vdsm_failures.log.gz?dl=1
It is possible that the multipath configuration I suggested is not
optimized correctly for your server or it is too old (last update on 2013).
Or you have some issues in the network or storage server.
I would continue with the storage vendor.
Nir
8:15 p.m.
New subject: oVirt Instability with Dell Compellent via iSCSI/Multipath
----- Original Message -----
From: "Chris Jones - BookIt.com Systems Administrator"
<chris.jones(a)bookit.com>
To: users(a)ovirt.org
Sent: Thursday, May 21, 2015 10:49:23 PM
Subject: Re: [ovirt-users] oVirt Instability with Dell Compellent via iSCSI/Multipath
I've applied the multipath.conf and iscsi.conf changes you recommended.
It seems to be running better. I was able to bring up all the hosts and
VMs without it falling apart.
I'm still seeing the domain "in problem" and "recovered" from
problem
warnings in engine.log, though. They were happening only when hosts were
activating and when I was mass launching many VMs. Is this normal?
2015-05-21 15:31:32,264 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-13) domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
blade6c2.ism.ld
2015-05-21 15:31:47,468 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-4) Domain
c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
vds: blade6c2.ism.ld
Here's the vdsm log from a node the engine was warning about
https://www.dropbox.com/s/yaubaxax1w499f1/vdsm2.log.gz?dl=1. It's
trimmed to just before and after it happened.
What is that repo stat command from your previous email, Nir? "repostat
vdsm.log" I don't see it on the engine or the node. Is it used to parse
the log? Where can I find it?
It is available here:
https://gerrit.ovirt.org/38749
Nir
12:57 p.m.
Hi Chris,
I have an Ovirt + Dell Compellent similar to yours (previous model, not
SC8000) and sometimes I faced issues similar to yours.
From my experience I can advise you to
A) check links between
SAN and servers, all paths, all configuration,
cabling. Everything should be setup correctly (all redundant paths green,
server mappings etc) BEFORE installing ovirt. We had a running KVM
environment before "upgrading" it to ovirt 3.5.1
B) Also check fencing is working both manually and automatically
(connections to iDRAC etc). This is a kind of pre-requisite to have HA
working.
C) I also noticed that when something is not going well on one of the
shared storage, this brings down the whole cluster (VM run, but a lot of
headaches being). First of all note that ovirt tries to stabilize the
situation itself for as long as ~15 minutes or more. It is slow in
re-fencing etc. Sometimes it enters in a loop and you have to locate the
problematic storage. You want to check the multipath on every server is
working correctly.
If you are having problems with just two nodes, I guess something is not
really ok at configuration level. I have 2 clusters, 12 hosts and several
(lots) of shared storage working and usually when something goes wrong is
because of an human error (like when I deleted the LUN on the SAN before
destroying the storage on the ovirt interface).
On the hand, I have the overall impression that the system is not
forgiving at all and that it is far from being rock solid.
Cheers
AG
10:29 p.m.
Sorry for the delay on this. I am in the process of reproducing the
error to get the logs.
On 05/19/2015 07:31 PM, Douglas Schilling Landgraf wrote:
Hello Chris,
On 05/19/2015 06:19 PM, Chris Jones - BookIt.com Systems Administrator
wrote:
> Engine: oVirt Engine Version: 3.5.2-1.el7.centos
> Nodes: oVirt Node - 3.5 - 0.999.201504280931.el7.centos
> Remote storage: Dell Compellent SC8000
> Storage setup: 2 nics connected to the Compellent. Several domains
> backed by LUNs. Several VM disk using direct LUN.
> Networking: Dell 10 Gb/s switches
>
> I've been struggling with oVirt completely falling apart due to storage
> related issues. By falling apart I mean most to all of the nodes
> suddenly losing contact with the storage domains. This results in an
> endless loop of the VMs on the failed nodes trying to be migrated and
> remigrated as the nodes flap between response and unresponsive. During
> these times, engine.log looks like this.
>
> 2015-05-19 03:09:42,443 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-50) domain
> c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 in problem. vds:
> blade6c1.ism.ld
> 2015-05-19 03:09:42,560 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) domain
> 0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 in problem. vds:
> blade2c1.ism.ld
> 2015-05-19 03:09:45,497 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-24) domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 in problem. vds:
> blade3c2.ism.ld
> 2015-05-19 03:09:51,713 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-46) domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
> blade4c2.ism.ld
> 2015-05-19 03:09:57,647 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-13) Domain
> c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
> vds: blade6c1.ism.ld
> 2015-05-19 03:09:57,782 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-6) domain
> 26929b89-d1ca-4718-90d6-b3a6da585451:generic_data_1 in problem. vds:
> blade2c1.ism.ld
> 2015-05-19 03:09:57,783 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-6) Domain
> 0b1d36e4-7992-43c7-8ac0-740f7c2cadb7:ovirttest1 recovered from problem.
> vds: blade2c1.ism.ld
> 2015-05-19 03:10:00,639 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-31) Domain
> c46adffc-614a-4fa2-9d2d-954f174f6a39:db_binlog_1 recovered from problem.
> vds: blade4c1.ism.ld
> 2015-05-19 03:10:00,703 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-17) domain
> 64101f40-0f10-471d-9f5f-44591f9e087d:logging_1 in problem. vds:
> blade1c1.ism.ld
> 2015-05-19 03:10:00,712 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-4) Domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
> vds: blade3c2.ism.ld
> 2015-05-19 03:10:06,931 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 recovered from problem.
> vds: blade4c2.ism.ld
> 2015-05-19 03:10:06,931 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> 05c8fa9c-fcbf-4a17-a3c6-011696a1b9a2:ovirttest2 has recovered from
> problem. No active host in the DC is reporting it as problematic, so
> clearing the domain recovery timer.
> 2015-05-19 03:10:06,932 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 recovered from problem.
> vds: blade4c2.ism.ld
> 2015-05-19 03:10:06,933 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-48) Domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 has recovered from
> problem. No active host in the DC is reporting it as problematic, so
> clearing the domain recovery timer.
> 2015-05-19 03:10:09,929 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-16) domain
> b050c455-5ab1-4107-b055-bfcc811195fc:os_data_1 in problem. vds:
> blade3c1.ism.ld
>
>
> My troubleshooting steps so far:
>
> 1. Tailing engine.log for "in problem" and "recovered from
problem"
> 2. Shutting down all the VMs.
> 3. Shutting down all but one node.
> 4. Bringing up one node at a time to see what the log reports.
vdsm.log in the node side, will help here too.
> When only one node is active everything is fine. When a second node
> comes up, I begin to see the log output as shown above. I've been
> struggling with this for over a month. I'm sure others have used oVirt
> with a Compellent and encountered (and worked around) similar problems.
> I'm looking for some help in figuring out if it's oVirt or something
> that I'm doing wrong.
>
> We're close to giving up on oVirt completely because of this.
>
> P.S.
>
> I've tested via bare metal and Proxmox with the Compellent. Not at the
> same scale but it seems to work fine there.
Do you mind to share your vdsm.log from the oVirt Node machine?
To go to console in oVirt Node, press F2 in TUI.
Files: /var/log/vdsm/vdsm*.log
# rpm -qa | grep -i vdsm
might help too.
Thanks!
9:11 p.m.
Since this thread shows up at the top of the search "oVirt compellent",
I should mention that this has been solved. The problem was a bad disk
in the Compellent's tier 2 storage. The mutlipath.conf and iscsi.conf
advice is still valid, though, and made oVirt more resilient when the
Compellent was struggling.
--
This email was Virus checked by UTM 9. For issues please contact the Windows Systems
Admin.
3257
days inactive
3271
days old
20 comments
7 participants
participants (7)
-
Andrea Ghelardi -
Chris Jones - BookIt.com Systems Administrator -
Daniel Helgenberger -
Douglas Schilling Landgraf -
Fabian Deutsch -
Jorick Astrego -
Nir Soffer