problem with iSCSI/multipath.
by Giorgio Bersano
Hi all.
We have an oVirt cluster in production happily running from the
beginning of 2014.
It started as 3.3 beta and now is Version 3.4.4-1.el6 .
Shared storage provided by an HP P2000 G3 iSCSI MSA.
The storage server is fully redundant (2 controllers, dual port disks,
4 iscsi connections per controller) and so is the connectivity (two
switches, multiple ethernet cards per server).
>From now on lets only talk about iSCSI connectivity.
The two oldest server have 2 nics each; they have been configured "by
hand" setting routes aimed to reach every scsi target from every nic.
On the "new" server we installed ovirt 3.5 to have a look at the
network configuration provided by oVirt.
In Data Center -> iSCSI Multipathing we defined an iSCSI Bond binding
together 3 server's nics and the 8 nics of the MSA.
The result is a system that has been functioning for months.
Recently we had to do an upgrade of the storage firmware.
This activity uploads the firmware to one of the MSA controllers then
reboots it. Being successful this is repeated on the other controller.
There is an impact on the I/O performance but there should be no
problems as every "volume" on the MSA remains visible on other paths.
Well, that's the theory.
On the two "hand configured" hosts we had no significant problems.
On the 3.5 host VMs started to migrate due to storage problems then
the situation got worse and it took more than an hour to bring again
the system to a good operating level.
I am inclined to believe that the culprit is the server's routing
table. Seems to me that the oVirt generated one is too simplistic and
prone to problems in case of connectivity loss (as in our situation or
when you have to reboot one of the switches).
Anyone on this list with strong experience on similar setup?
I have included below some background information.
I'm available to provide anything useful to further investigate the case.
TIA,
Giorgio.
-------------------
context information
-------------------
oVirt Compatibility Version: 3.4
two FUJITSU PRIMERGY RX300 S5 hosts
CPU: Intel(R) Xeon(R) E5504 @ 2.00GHz / Intel Nehalem Family
OS Version: RHEL - 6 - 6.el6.centos.12.2
Kernel Version: 2.6.32 - 504.16.2.el6.x86_64
KVM Version: 0.12.1.2 - 2.448.el6_6.2
LIBVIRT Version: libvirt-0.10.2-46.el6_6.6
VDSM Version: vdsm-4.14.17-0.el6
RAM: 40GB
mom-0.4.3-1.el6.noarch.rpm
ovirt-release34-1.0.3-1.noarch.rpm
qemu-img-rhev-0.12.1.2-2.448.el6_6.2.x86_64.rpm
qemu-kvm-rhev-0.12.1.2-2.448.el6_6.2.x86_64.rpm
qemu-kvm-rhev-tools-0.12.1.2-2.448.el6_6.2.x86_64.rpm
vdsm-4.14.17-0.el6.x86_64.rpm
vdsm-cli-4.14.17-0.el6.noarch.rpm
vdsm-hook-hostusb-4.14.17-0.el6.noarch.rpm
vdsm-hook-macspoof-4.14.17-0.el6.noarch.rpm
vdsm-python-4.14.17-0.el6.x86_64.rpm
vdsm-python-zombiereaper-4.14.17-0.el6.noarch.rpm
vdsm-xmlrpc-4.14.17-0.el6.noarch.rpm
# ip route list table all |grep 192.168.126.
192.168.126.87 dev eth4 table 4 proto kernel scope link src 192.168.126.65
192.168.126.86 dev eth4 table 4 proto kernel scope link src 192.168.126.65
192.168.126.81 dev eth4 table 4 proto kernel scope link src 192.168.126.65
192.168.126.80 dev eth4 table 4 proto kernel scope link src 192.168.126.65
192.168.126.77 dev eth4 table 4 proto kernel scope link src 192.168.126.65
192.168.126.0/24 dev eth4 table 4 proto kernel scope link src 192.168.126.65
192.168.126.0/24 dev eth3 proto kernel scope link src 192.168.126.64
192.168.126.0/24 dev eth4 proto kernel scope link src 192.168.126.65
192.168.126.85 dev eth3 table 3 proto kernel scope link src 192.168.126.64
192.168.126.84 dev eth3 table 3 proto kernel scope link src 192.168.126.64
192.168.126.83 dev eth3 table 3 proto kernel scope link src 192.168.126.64
192.168.126.82 dev eth3 table 3 proto kernel scope link src 192.168.126.64
192.168.126.76 dev eth3 table 3 proto kernel scope link src 192.168.126.64
192.168.126.0/24 dev eth3 table 3 proto kernel scope link src 192.168.126.64
broadcast 192.168.126.0 dev eth3 table local proto kernel scope
link src 192.168.126.64
broadcast 192.168.126.0 dev eth4 table local proto kernel scope
link src 192.168.126.65
local 192.168.126.65 dev eth4 table local proto kernel scope host
src 192.168.126.65
local 192.168.126.64 dev eth3 table local proto kernel scope host
src 192.168.126.64
broadcast 192.168.126.255 dev eth3 table local proto kernel scope
link src 192.168.126.64
broadcast 192.168.126.255 dev eth4 table local proto kernel scope
link src 192.168.126.65
one HP ProLiant DL560 Gen8 host
CPU: Intel(R) Xeon(R) CPU E5-4610 v2 @ 2.30GHz / Intel SandyBridge Family
OS Version:RHEL - 6 - 6.el6.centos.12.2
Kernel Version: 2.6.32 - 504.16.2.el6.x86_64
KVM Version: 0.12.1.2 - 2.448.el6_6.2
LIBVIRT Version: libvirt-0.10.2-46.el6_6.6
VDSM Version: vdsm-4.16.14-0.el6
RAM: 256GB
mom-0.4.3-1.el6.noarch.rpm
ovirt-release35-002-1.noarch.rpm
qemu-img-rhev-0.12.1.2-2.448.el6_6.2.x86_64.rpm
qemu-kvm-rhev-0.12.1.2-2.448.el6_6.2.x86_64.rpm
qemu-kvm-rhev-tools-0.12.1.2-2.448.el6_6.2.x86_64.rpm
vdsm-4.16.14-0.el6.x86_64.rpm
vdsm-cli-4.16.14-0.el6.noarch.rpm
vdsm-hook-hostusb-4.16.14-0.el6.noarch.rpm
vdsm-hook-macspoof-4.16.14-0.el6.noarch.rpm
vdsm-jsonrpc-4.16.14-0.el6.noarch.rpm
vdsm-python-4.16.14-0.el6.noarch.rpm
vdsm-python-zombiereaper-4.16.14-0.el6.noarch.rpm
vdsm-xmlrpc-4.16.14-0.el6.noarch.rpm
vdsm-yajsonrpc-4.16.14-0.el6.noarch.rpm
# ip route list table all |grep 192.168.126.
192.168.126.0/24 dev p6p1 proto kernel scope link src 192.168.126.34
192.168.126.0/24 dev p3p1 proto kernel scope link src 192.168.126.33
192.168.126.0/24 dev em3 proto kernel scope link src 192.168.126.32
local 192.168.126.32 dev em3 table local proto kernel scope host
src 192.168.126.32
local 192.168.126.33 dev p3p1 table local proto kernel scope host
src 192.168.126.33
broadcast 192.168.126.0 dev p6p1 table local proto kernel scope
link src 192.168.126.34
broadcast 192.168.126.0 dev p3p1 table local proto kernel scope
link src 192.168.126.33
broadcast 192.168.126.0 dev em3 table local proto kernel scope link
src 192.168.126.32
local 192.168.126.34 dev p6p1 table local proto kernel scope host
src 192.168.126.34
broadcast 192.168.126.255 dev p6p1 table local proto kernel scope
link src 192.168.126.34
broadcast 192.168.126.255 dev p3p1 table local proto kernel scope
link src 192.168.126.33
broadcast 192.168.126.255 dev em3 table local proto kernel scope
link src 192.168.126.32
9 years, 3 months
ovirt 3.5.2 issues with nodes becoming "Non Operational"
by Chris Liebman
Hi,
I'm new to oVirt and recently built a 10 node ovirt 3.5 DC with shared
storage using gluster configured as distributed-replicated (replication =
2). Shortly after 7 of the 10 nodes dropped, one at a time over a few
hours, into "Non Operational" state. Attempting to activate one of these
nodes gives the error: "Failed to connect Host ovirt-node260 to Storage
Pool LADC-TBX". Attempting to put the node into Maintenance eaves the node
stuck in "Preparing For maintenance".
When I rebooted one of the nodes I see this in the nodes event list:
"Host ovirt-node269 reports about one of the Active Storage Domains as
Problematic."
I see many of these errors in the vdsm log from the failed nodes:
Thread-10000::ERROR::2015-08-12
> 10:01:17,748::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
>
> Traceback (most recent call last):
>
> File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py", line 501,
> in _serveRequest
>
> res = method(**params)
>
> File "/usr/share/vdsm/rpc/Bridge.py", line 267, in _dynamicMethod
>
> result = fn(*methodArgs)
>
> File "/usr/share/vdsm/API.py", line 1330, in getStats
>
> stats.update(self._cif.mom.getKsmStats())
>
> File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
>
> stats = self._mom.getStatistics()['host']
>
> File "/usr/lib/python2.6/site-packages/mom/MOMFuncs.py", line 75, in
> getStatistics
>
> host_stats = self.threads['host_monitor'].interrogate().statistics[-1]
>
> AttributeError: 'NoneType' object has no attribute 'statistics'
>
Any help here is appreciated.
-- Chris
9 years, 3 months
Gluster Replica 3 using arbiter node?
by Adrian Lewis
Hi,
Just wondering if it will be possible to create gluster replica 3 volumes
in oVirt 3.6 that use the arbiter function instead of actually storing
three copies of the data? If so, could this be used for the hosted engine
on gluster feature which from what I can tell requires replica 3?
Many thanks,
Adrian
9 years, 3 months
vgs and lvs return empty list
by like.ma@cs2c.com.cn
This is a multi-part message in MIME format.
------=_001_NextPart835358688211_=----
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: base64
SGksIGFsbA0KDQpJIGhhdmUgYSBvVmlydDMuNCBlbnZpcm9ubWVudC4gSSBjcmVhdGUgYSBGQyBz
dG9yYWdlIGRvbWFpbiBhbmQgYSBkaXNrIG9uIHRoaXMgRkMgc3RvcmFnZSBkb21haW4uIEV2ZXJ5
dGhpbmcgd29ya3Mgd2VsbC4gQnV0IGkgaGF2ZSBhIHF1ZXN0aW9uOg0KV2hlbiBpIGxvZ2luIHRo
ZSB2ZHNtIGhvc3QsIGFuZCB1c2UgdGhlIHZncy9sdnMgY29tbWFuZCB0byBzaG93IHRoZSB2ZyB3
aGljaCByZXByZXNlbnQgRkMgc3RvcmFnZSBkb21haW4gYW5kIHRoZSBsdiB3aGljaCByZXByZXNl
bnQgdGhlIGRpc2sgb24gdGhlIEZDIHN0b3JhZ2UgZG9tYWluLCB0aGUgY29tbWFuZCByZXR1cmVu
cyBlbXB0eSByZXN1bHQgbGlzdC4gSXQgc2VlbXMgbGlrZSB0aGVyZSBpcyBubyB2ZyBvciBsdiBh
dCBhbGwuIEJ1dCBpZiBpIHVzZSB0aGUgbHZjaGFuZ2UgLWF5IDxsdiBwYXRoPiB0byBhY3RpdmF0
ZSB0aGUgbHYgb24gdGhlIGhvc3QsIHRoZW4gdGhlIHZncy9sdnMgY29tbWFuZCByZXR1cm5zIHRo
ZSBleHBlY3RlZCB2Zy9sdiBsaXN0LiBJIHdhbnQgdG8ga25vdyB3aHkgaXQgaXMgbGlrZSB0aGlz
PyBBZnRlciBhbGwsIGlmIGkgdXNlIHZnY3JlYXRlIGNvbW1hbmQgdG8gY3JlYXRlIGEgbmV3IHZn
LCB0aGVuIGkgY2FuIHVzZSB2Z3MgY29tbWFuZCB0byBzaG93IHRoZSB2ZyBsaXN0IGltbWVkaWF0
ZWx5Lg0KDQpUaGFua3MsIGxvb2tpbmcgZm93YXJkIHRvIHlvdXIgcmVwbHkuDQoNCg0KDQpsaWtl
Lm1hQGNzMmMuY29tLmNuDQo=
------=_001_NextPart835358688211_=----
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse=
t=3Dus-ascii"><style>body { line-height: 1.5; }body { font-size: 10.5pt; f=
ont-family: ????; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><=
body>=0A<div><span></span>Hi, all</div><div><br></div><div>I have a oVirt3=
.4 environment. I create a FC storage domain and a disk on this FC storage=
domain. Everything works well. But i have a question:</div><div>When i lo=
gin the vdsm host, and use the vgs/lvs command to show the vg which repres=
ent FC storage domain and the lv which represent the disk on the FC storag=
e domain, the command returens empty result list. It seems like there is n=
o vg or lv at all. But if i use the lvchange -ay <lv path> to activa=
te the lv on the host, then the vgs/lvs command returns the expected vg/lv=
list. I want to know why it is like this? After all, if i use vgcreate co=
mmand to create a new vg, then i can use vgs command to show the vg list i=
mme<span style=3D"font-size: 10.5pt; line-height: 1.5; background-color: w=
indow;">diately.</span></div><div><span style=3D"font-size: 10.5pt; line-h=
eight: 1.5; background-color: window;"><br></span></div><div><span style=
=3D"font-size: 10.5pt; line-height: 1.5; background-color: window;">Thanks=
, looking foward to your reply.</span></div>=0A<div><br></div><hr style=3D=
"width: 210px; height: 1px;" color=3D"#b5c4df" size=3D"1" align=3D"left">=
=0A<div><span><div style=3D"MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE:=
10pt"><div>like.ma(a)cs2c.com.cn</div></div></span></div>=0A</body></html>
------=_001_NextPart835358688211_=------
9 years, 3 months
CPU Threads Help!!!
by zhangjian2011
--------------020500020201060207010309
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Hi, Guys
Recently I am investigating the Optimization Policy in Cluster and try
to use “CPU Threads”.
As the manual of description:
========
For example, a 24-core
system with 2 threads per core (48
threads total) can run virtual machines with
up to 48 cores each
========
My host CPU is
i3-2120 CPU (2 cores 4 threads)
So I think the expect result case is :
1. If the “Count Threads As Cores” *disable*, we can’t run virtual vm
with above 2 vCPU
2. If the “Count Threads As Cores” *enable*, we can’t run virtual vm
with above 4 vCPU
Now the problem is about result case 1:
1. When I set “Count Threads As Cores” *disable*, I still can run
virtual vm with 4 vCPU
Can anyone help me to explain it ???
Regards,
Jian
--------------020500020201060207010309
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p class="MsoNormal"><span lang="EN-US">Hi, Guys<o:p></o:p></span></p>
<span lang="EN-US">Recently I am investigating the Optimization
Policy in Cluster and try to use “</span><span
style="font-family:"Arial","sans-serif";color:black;background:white"
lang="EN-US">CPU Threads</span><span lang="EN-US">”.<o:p></o:p></span>
<p class="MsoNormal"><span lang="EN-US">As the manual of
description:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">========<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:left;text-autospace:none"
align="left"><span style="font-family:DejaVuSans" lang="EN-US">For
example, a 24-core<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:left;text-autospace:none"
align="left"><span style="font-family:DejaVuSans" lang="EN-US">system
with 2 threads per core (48<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:left;text-autospace:none"
align="left"><span style="font-family:DejaVuSans" lang="EN-US">threads
total) can run virtual machines with<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:DejaVuSans"
lang="EN-US">up to 48 cores each</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">========<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">My host CPU is<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">i3-2120 CPU (2 cores 4
threads)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So I think the expect result
case is :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1. If the “</span><span
style="font-family:"Arial Unicode
MS","sans-serif";color:black;background:white"
lang="EN-US">Count Threads As Cores</span><span lang="EN-US">”
</span><b><span style="font-size:14.0pt" lang="EN-US">disable</span></b><span
lang="EN-US">, we can’t run virtual vm with above 2 vCPU<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2. If the “</span><span
style="font-family:"Arial Unicode
MS","sans-serif";color:black;background:white"
lang="EN-US">Count Threads As Cores</span><span lang="EN-US">”
</span><b><span style="font-size:14.0pt" lang="EN-US">enable</span></b><span
lang="EN-US">, we can’t run virtual vm with above 4 vCPU<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Now the problem is about
result case 1:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1. When I set “</span><span
style="font-family:"Arial Unicode
MS","sans-serif";color:black;background:white"
lang="EN-US">Count Threads As Cores</span><span lang="EN-US">”
</span><b><span style="font-size:14.0pt" lang="EN-US">disable</span></b><span
lang="EN-US">, I still can run virtual vm with 4 vCPU<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<span lang="EN-US">Can anyone help me to explain it ???<br>
<br>
Regards,<br>
Jian<br>
</span>
</body>
</html>
--------------020500020201060207010309--
9 years, 3 months
Stuck installing Neutron Appliance
by Alan Murrell
Hello.
I am running oVirt 3.5.1. on a single server running Hosted Engine. I
am running through
<http://www.ovirt.org/Features/NeutronVirtualAppliance#Run_the_neutron_ser...> and am stuck on the section "Install a Host with the network provider". Specifically, I am stuck on Step 2, which
is:
2. Install the host with the external network provider by clicking
on the 'Network Provider' left tab
I am not sure where to click. In my "Tree", I do have "External
providers", but I already added my Neutrom Provider in the previous
section, 'Configure Neutron network provider on ovirt-engine'. Is
there somewhere else I need to add this, or because I am running a
self-hosted engine, it ends up being the same thing?
Also in this same section, Step 7 where I am to run 'ovs-vsctl
add-port br-neutron neutron', I don't have this command ('ovs-vsctl'
doesn't seem to be installed, which I believe is part of openVswtich).
I didn't see any steps beforehand that would have installed
openVswitch. Is there a step missing?
Thanks! :-)
Regards,
Alan
9 years, 3 months
iSCSI question... LUNS->Targets balnk
by Alan Murrell
Hello,
I am hoping someone here has had experience in setting up an iSCSI
target using 'targetcli'. I followed the following guide:
<http://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persisten...>. This is on my single host (with hosted engine currently running; I am using a seperate HDD for the iSCSI
storage)
The iSCSI initiator of my host, from it's information page in oVirt,
is <iqn.1994-05.com.redhat:ba4cc8b3368e>, so I created an ACL with
that name. Here is a result of the listing in the 'acls' directory
for my IQN:
--- START ---
/iscsi/iqn.20...gt1/tpg1/acls> ls
o- acls ..............................................................
[ACLs: 1]
o- iqn.1994-05.com.redhat:ba4cc8b3368e ......................
[Mapped LUNs: 1]
| o- mapped_lun0 .................................... [lun0
block/block1 (rw)]
--- END ---
So it appears as though my host's initiator is mapped to my LUN 0.
When I go in to oVirt and add iSCSI storage, the "Discover" finds my
target, and I can even log in successfully, however when I click on
the "LUNS->Targets" side tab, there are no LUNs listed, so I cannot
add any iSCSI storage.
Any ideas on why the LUN is not showing up?
Thanks! :-)
Regards,
Alan
9 years, 3 months
Multiple Networks on Self Hosted Engine
by Brent Miles
I am trying to install RHEVM 3.5 Self Hosted Engine. I have multiple vlan
interfaces on the host that I would also like to have available on the
self hosted engine. I have eth0.305 and eth3.306 on the host that I would
like to have available on RHEVM. During the hosted-engine setup I get the
option to add only one of interfaces but not both and do not seem to be
able to add a second network at any point during setup or after the engine
vm is installed. Any advise on doing this?
9 years, 3 months
[Reports] Virtual machine Network usage statistics
by Lionel Caignec
Hi,
i've recently installed ovirt-engine-report, and i've a little problem with reports about VM.
All graphs from "Network inteface usage" for VM are empty.
Is it some configuration to do to get this data?
All data for cpu/memory are ok.
--
Lionel
9 years, 3 months