5:43 p.m.
--=_96b09eee02d72544a4bd8a6df3818a5b
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII
Hi
Recently, i built a small oVirt platform with 2 dedicated servers and
GlusterFS to synch the VM storage.
oVirt Setup is simple:
ovirt01 : Host Agent (VDSM) + oVirt Engine
ovirt02 : Host Agent (VDSM)
Version :
ovirt-release35-005-1.noarch
ovirt-engine-3.5.4.2-1.el7.centos.noarch
vdsm-4.16.26-0.el7.centos.x86_64
vdsm-gluster-4.16.26-0.el7.centos.noarch
glusterfs-server-3.7.4-2.el7.x86_64
GlusterFS Setup is simple, 2 bricks in replicate mode.
It was done in shell; not using oVirt GUI, and then it was added in
STORAGE as a new DOMAIN; Type:DATA GlusterFS V3
# gluster volume info
Volume Name: ovirt
Type: Replicate
Volume ID: 043d2d36-dc2c-4f75-9d28-96dbac25d07c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: ovirt01:/gluster/ovirt
Brick2: ovirt02:/gluster/ovirt
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: true
auth.allow: IP_A, IP_B
network.ping-timeout: 10
storage.owner-uid: 36
storage.owner-gid: 36
server.allow-insecure: on
the data are reachable on the 2 nodes through a moint point that oVirt
created when i configured the Storage with the GUI:
localhost:/ovirt 306G 216G 78G 74%
/rhev/data-center/mnt/glusterSD/localhost:_ovirt
I created 7 VM on this shared storage and all is working fine. I can do
Live migration; all is working.
But when i check /var/log/ovirt/engine.log on ovirt01, there are error
in loop every 2 seconds:
2015-10-11 17:29:50,971 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-29) [34dbe5cf] START,
GlusterVolumesListVDSCommand(HostName = ovirt02, HostId =
65a5bb5d-721f-4a4b-9e77-c4b9162c0aa6), log id: 41443b77
2015-10-11 17:29:50,998 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-29) [34dbe5cf] Could not add brick
ovirt02:/gluster/ovirt to volume 043d2d36-dc2c-4f75-9d28-96dbac25d07c -
server uuid 3c340e59-334f-4aa6-ad61-af2acaf3cad6 not found in cluster
fb976d4f-de13-449b-93e8-600fcb59d4e6
2015-10-11 17:29:50,999 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-29) [34dbe5cf] FINISH,
GlusterVolumesListVDSCommand, return:
{043d2d36-dc2c-4f75-9d28-96dbac25d07c=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@200ae0d1},
log id: 41443b77
2015-10-11 17:29:51,001 WARN
[org.ovirt.engine.core.bll.gluster.GlusterSyncJob]
(DefaultQuartzScheduler_Worker-29) [34dbe5cf] Volume ovirt contains a
apparently corrupt brick(s). Hence will not add it to engine at this
point.
I played a lot with oVirt at first it was running on a single node; in
Local Datacenter; then i added a second node, move the first host to a
new datacenter; migrated the images VM etc; with some pain at some very
moment and now all looks fine but i prefer to double check.
So, i want to know if there is a real issue with ovirt/gluster setup
that i don't see, any info are welcome because i'm a bit worried to see
these message in LOOP on the log.
Thanks in advance;
Regards
Nico
--=_96b09eee02d72544a4bd8a6df3818a5b
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=
=3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family:
Verdana,Gen=
eva,sans-serif' <p
class=3D"p1"><span class=3D"s1">Hi</span <p class=3D"p1"><span
class=3D"s1">Recently, i built a small oVirt platform=
with 2 dedicated servers and GlusterFS to synch the VM storage.</span=
<p class=3D"p1"><span
class=3D"s1">oVirt Setup is simple:</span <p
class=3D"p1"><span class=3D"s1">ovirt01 : Host
Agent (VDSM) =
+ oVirt Engine </span <p class=3D"p2"> </p <p class=3D"p1"><span
class=3D"s1">Version : </span <p
class=3D"p1"><span
class=3D"s1">ovirt-release35-005-1.noarch</span <p class=3D"p1"><span
class=3D"s1">vdsm-gluster-4.16.26-0.el7.centos.noarch=
</span <p class=3D"p2"> </p <p class=3D"p1"><span
class=3D"s1">GlusterFS Setup is simple, 2 bricks in r=
eplicate mode.</span <p
class=3D"p1"><span class=3D"s1"># gluster volume
info</span <p
class=3D"p1"><span class=3D"s1">Volume Name:
ovirt</span <p class=3D"p1"><span
class=3D"s1">Brick2: ovirt02:/gluster/ovirt</span></p=
<p class=3D"p1"><span
class=3D"s1">Options Reconfigured:</span <p
class=3D"p2"> </p <p
class=3D"p1"><span class=3D"s1">the data are reachable on the
2 nodes th=
rough a moint point that oVirt created when i configured the Storage with t=
he GUI:</span <p
class=3D"p1"><span class=3D"s1">localhost:/ovirt
&n=
bsp; 306G 216G
78G 74=
% /rhev/data-center/mnt/glusterSD/localhost:_ovirt</span <p
class=3D"p1"><span class=3D"s1">I created 7 VM on this shared
storage an=
d all is working fine. I can do Live migration; all is working.</span <p class=3D"p1"><span
class=3D"s1">But when i check /var/log/ovirt/engine=
=2Elog on ovirt01, there are error in loop every 2 seconds:</span <p class=3D"p1"><span
class=3D"s1">2015-10-11 17:29:50,971 INFO [org=
=2Eovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (Defau=
ltQuartzScheduler_Worker-29) [34dbe5cf] START, GlusterVolumesListVDSCommand=
(HostName =3D ovirt02, HostId =3D 65a5bb5d-721f-4a4b-9e77-c4b9162c0aa6), lo=
g id: 41443b77</span <p
class=3D"p2"><span class=3D"s1"></span>I played a lot
with oVirt at firs=
t it was running on a single node; in Local Datacenter; then i added a seco=
nd node, move the first host to a new datacenter; migrated the images VM et=
c; with some pain at some very moment and now all looks fine but i prefer t=
o double check.</p <p
class=3D"p1"><span class=3D"s1">So, i want to know if there
is a real is=
sue with ovirt/gluster setup that i don’t see, any info are welcome b=
ecause i’m a bit worried to see these message in LOOP on the log.</sp=
an <p class=3D"p1"><span
class=3D"s1">Thanks in advance;</span <p
class=3D"p1"><span class=3D"s1">Regards</span
</body></html
--=_96b09eee02d72544a4bd8a6df3818a5b--
</p>
<p class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p
class=3D"p1"><span class=3D"s1">ovirt02 : Host
Agent (VDSM)</span>=
</p</p>
<p class=3D"p2"> </p</p>
<p class=3D"p1"><span
class=3D"s1">ovirt-engine-3.5.4.2-1.el7.centos.noarch=
</span</p>
<p
class=3D"p1"><span
class=3D"s1">vdsm-4.16.26-0.el7.centos.x86_64</span><=
/p</p>
<p
class=3D"p1"><span
class=3D"s1">glusterfs-server-3.7.4-2.el7.x86_64</spa=
n</p>
<p class=3D"p2"> </p</p>
<p
class=3D"p1"><span class=3D"s1">It was done in
shell; not using oV=
irt GUI, and then it was added in STORAGE as a new DOMAIN; Type:DATA Gluste=
rFS V3</span</p>
<p
class=3D"p2"> </p</p>
<p
class=3D"p2"> </p</p>
<p
class=3D"p1"><span class=3D"s1">Type: Replicate</span</p>
<p class=3D"p1"><span
class=3D"s1">Volume ID: 043d2d36-dc2c-4f75-9d28-96dba=
c25d07c</span</p>
<p
class=3D"p1"><span class=3D"s1">Status: Started</span</p>
<p class=3D"p1"><span
class=3D"s1">Number of Bricks: 1 x 2 =3D 2</span</p>
<p class=3D"p1"><span
class=3D"s1">Transport-type: tcp</span</p>
<p class=3D"p1"><span
class=3D"s1">Bricks:</span</p>
<p class=3D"p1"><span class=3D"s1">Brick1:
ovirt01:/gluster/ovirt</span></p=
</p>
<p class=3D"p1"><span
class=3D"s1">performance.readdir-ahead: on</span</p>
<p class=3D"p1"><span
class=3D"s1">nfs.disable: true</span</p>
<p class=3D"p1"><span class=3D"s1">auth.allow: IP_A,
IP_B</span</p>
<p
class=3D"p1"><span class=3D"s1">network.ping-timeout:
10</span</p>
<p
class=3D"p1"><span class=3D"s1">storage.owner-uid:
36</span</p>
<p
class=3D"p1"><span class=3D"s1">storage.owner-gid:
36</span</p>
<p
class=3D"p1"><span class=3D"s1">server.allow-insecure:
on</span</p>
<p
class=3D"p2"> </p</p>
<p
class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p
class=3D"p1"><span class=3D"s1">2015-10-11 17:29:50,998 WARN
[org=
=2Eovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (=
DefaultQuartzScheduler_Worker-29) [34dbe5cf] Could not add brick ovirt02:/g=
luster/ovirt to volume 043d2d36-dc2c-4f75-9d28-96dbac25d07c - server uuid 3=
c340e59-334f-4aa6-ad61-af2acaf3cad6 not found in cluster fb976d4f-de13-449b=
-93e8-600fcb59d4e6</span</p>
<p
class=3D"p1"><span class=3D"s1">2015-10-11 17:29:50,999 INFO
[org=
=2Eovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (Defau=
ltQuartzScheduler_Worker-29) [34dbe5cf] FINISH, GlusterVolumesListVDSComman=
d, return: {043d2d36-dc2c-4f75-9d28-96dbac25d07c=3Dorg.ovirt.engine.core.co=
mmon.businessentities.gluster.GlusterVolumeEntity@200ae0d1}, log id: 41443b=
77</span</p>
<p
class=3D"p1"><span class=3D"s1">2015-10-11 17:29:51,001 WARN
[org=
=2Eovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler_Wo=
rker-29) [34dbe5cf] Volume ovirt contains a apparently corrupt brick(s). He=
nce will not add it to engine at this point.</span</p>
<p class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p class=3D"p2"> </p</p>
<p class=3D"p1"><span
class=3D"s1">Nico</span</p>
<div> </div
9:59 a.m.
On Sun, Oct 11, 2015 at 6:43 PM, Nico <gluster(a)distran.org> wrote:
Recently, i built a small oVirt platform with 2 dedicated servers and
GlusterFS to synch the VM storage.
Bricks:
Brick1: ovirt01:/gluster/ovirt
Brick2: ovirt02:/gluster/ovirt
This looks like replica 2 - this is not supported.
You can use either replica 1 (testing) or replica 3 (production).
But when i check /var/log/ovirt/engine.log on ovirt01, there are
error in loop every 2 seconds:
To understand such error we need to see the vdsm
log.
Nir
10:14 a.m.
--=_d69a1f5c639decb7303d9c14e20f8797
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
Le 2015-10-12 09:59, Nir Soffer a écrit :
state preparing
Thread-55245::INFO::2015-10-12
10:12:22,530::logUtils::44::dispatcher::(wrapper) Run and protect:
getVolumeSize(sdUUID='d44ee4b0-8d36-467a-9610-c682a618b698',
spUUID='0ae7120a-430d-4534-9a7e-59c53fb2e804',
imgUUID='3454b077-297b-4b89-b8ce-a77f6ec5d22e',
volUUID='933da0b6-6a05-4e64-958a-e1c030cf5ddb', options=None)
Thread-55245::INFO::2015-10-12
10:12:22,535::logUtils::47::dispatcher::(wrapper) Run and protect:
getVolumeSize, Return response: {'truesize': '158983839744',
'apparentsize': '161061273600'}
Thread-55245::DEBUG::2015-10-12
10:12:22,535::task::1191::Storage.TaskManager.Task::(prepare)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::finished: {'truesize':
'158983839744', 'apparentsize': '161061273600'}
Thread-55245::DEBUG::2015-10-12
10:12:22,535::task::595::Storage.TaskManager.Task::(_updateState)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::moving from state preparing
-> state finished
Thread-55245::DEBUG::2015-10-12
10:12:22,535::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-55245::DEBUG::2015-10-12
10:12:22,536::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-55245::DEBUG::2015-10-12
10:12:22,536::task::993::Storage.TaskManager.Task::(_decref)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::ref 0 aborting False
Thread-55245::DEBUG::2015-10-12
10:12:22,545::libvirtconnection::143::root::(wrapper) Unknown
libvirterror: ecode: 80 edom: 20 level: 2 message: metadata not found:
Requested metadata element is not present
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:23,138::stompReactor::98::Broker.StompAdapter::(handle_frame)
Handling message <StompFrame command='SEND'
JsonRpcServer::DEBUG::2015-10-12
10:12:23,139::__init__::530::jsonrpc.JsonRpcServer::(serve_requests)
Waiting for request
Thread-167406::DEBUG::2015-10-12
10:12:23,142::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
Thread-37810::DEBUG::2015-10-12
10:12:24,194::fileSD::262::Storage.Misc.excCmd::(getReadDelay)
/usr/bin/dd
if=/rhev/data-center/mnt/ovirt01:_data_iso/5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)
Thread-37810::DEBUG::2015-10-12
10:12:24,201::fileSD::262::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
<err> = '0+1 records in\n0+1 records out\n317 bytes (317 B) copied,
0.000131729 s, 2.4 MB/s\n'; <rc> = 0
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:26,148::stompReactor::98::Broker.StompAdapter::(handle_frame)
Handling message <StompFrame command='SEND'
JsonRpcServer::DEBUG::2015-10-12
10:12:26,149::__init__::530::jsonrpc.JsonRpcServer::(serve_requests)
Waiting for request
Thread-167407::DEBUG::2015-10-12
10:12:26,151::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
VM Channels Listener::DEBUG::2015-10-12
10:12:26,972::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno
35.
Thread-30::DEBUG::2015-10-12
10:12:28,358::fileSD::262::Storage.Misc.excCmd::(getReadDelay)
/usr/bin/dd
if=/rhev/data-center/mnt/glusterSD/localhost:_ovirt/d44ee4b0-8d36-467a-9610-c682a618b698/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)
Thread-30::DEBUG::2015-10-12
10:12:28,451::fileSD::262::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
<err> = '0+1 records in\n0+1 records out\n470 bytes (470 B) copied,
0.000152738 s, 3.1 MB/s\n'; <rc> = 0
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:29,157::stompReactor::98::Broker.StompAdapter::(handle_frame)
Handling message <StompFrame command='SEND'
JsonRpcServer::DEBUG::2015-10-12
10:12:29,252::__init__::530::jsonrpc.JsonRpcServer::(serve_requests)
Waiting for request
Thread-167408::DEBUG::2015-10-12
10:12:29,254::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:32,260::stompReactor::98::Broker.StompAdapter::(handle_frame)
Handling message <StompFrame command='SEND'
JsonRpcServer::DEBUG::2015-10-12
10:12:32,262::__init__::530::jsonrpc.JsonRpcServer::(serve_requests)
Waiting for request
Thread-167409::DEBUG::2015-10-12
10:12:32,264::task::595::Storage.TaskManager.Task::(_updateState)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::moving from state init - state preparing
Thread-167409::INFO::2015-10-12
10:12:32,264::logUtils::44::dispatcher::(wrapper) Run and protect:
repoStats(options=None)
Thread-167409::INFO::2015-10-12
10:12:32,265::logUtils::47::dispatcher::(wrapper) Run and protect:
repoStats, Return response: {u'd44ee4b0-8d36-467a-9610-c682a618b698':
{'code': 0, 'actual': True, 'version': 3, 'acquired':
True, 'delay':
'0.000152738', 'lastCheck': '3.6', 'valid': True},
u'5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0': {'code': 0, 'actual':
True,
'version': 0, 'acquired': True, 'delay': '0.000131729',
'lastCheck':
'8.1', 'valid': True}}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::1191::Storage.TaskManager.Task::(prepare)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::finished:
{u'd44ee4b0-8d36-467a-9610-c682a618b698': {'code': 0, 'actual':
True,
'version': 3, 'acquired': True, 'delay': '0.000152738',
'lastCheck':
'3.6', 'valid': True}, u'5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0':
{'code':
0, 'actual': True, 'version': 0, 'acquired': True,
'delay':
'0.000131729', 'lastCheck': '8.1', 'valid': True}}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::595::Storage.TaskManager.Task::(_updateState)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::moving from state preparing
-> state finished
Thread-167409::DEBUG::2015-10-12
10:12:32,265::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::993::Storage.TaskManager.Task::(_decref)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::ref 0 aborting False
Thread-167409::DEBUG::2015-10-12
10:12:32,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:32,275::stompReactor::98::Broker.StompAdapter::(handle_frame)
Handling message <StompFrame command='SEND'
JsonRpcServer::DEBUG::2015-10-12
10:12:32,278::__init__::530::jsonrpc.JsonRpcServer::(serve_requests)
Waiting for request
Thread-167410::DEBUG::2015-10-12
10:12:32,283::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
--=_d69a1f5c639decb7303d9c14e20f8797
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=
=3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family:
Verdana,Gen=
eva,sans-serif' <p> </p <p>Le 2015-10-12 09:59, Nir Soffer a
écrit :</p <blockquote
type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2=
px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!--
meta ig=
nored -- <div class=3D"pre"
style=3D"margin: 0; padding: 0; font-family: monospace">=
On Sun, Oct 11, 2015 at 6:43 PM, Nico <<a href=3D"mailto:gluster@distran=
=2Eorg">gluster(a)distran.org</a>&gt; wrote:
<blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left:
#1010ff 2=
px solid; margin: 0">Recently, i built a small oVirt platform with 2 dedica=
ted servers and GlusterFS to synch the VM storage.<br /> Bricks:<br /> <br
=
/> Brick1: ovirt01:/gluster/ovirt<br /> <br /> Brick2:
ovirt02:/gluster/ovi=
rt</blockquote <br /> This looks like replica
2 - this is not supported.<br /> <br /> You =
can use either replica 1 (testing) or replica 3 (production).<br /> <br / <blockquote type=3D"cite" style=3D"padding: 0
0.4em; border-left: #1010ff 2=
px solid; margin: 0">But when i check /var/log/ovirt/engine.log on ovirt01,=
there are error in loop every 2 seconds:</blockquote To
understand such error we need to see the vdsm log.<br /> <br /> Nir</div=
</blockquote
<p>Yeah it is replica 2 as i've only 2 dedicated servers.</p <p>why are you saying it is not supported ?
Through oVirt GUI, it is p=
ossible to create a Gluster Volume with 2 bricks in repllcate mode; i tried=
it also.</p <p>here the last entries of
vdsm.log</p <p> </p <p>hread-167405::DEBUG::2015-10-12
10:12:20,132::stompReactor::163::yajsonr=
pc.StompServer::(send) Sending response<br />Thread-55245::DEBUG::2015-10-1=
2 10:12:22,529::task::595::Storage.TaskManager.Task::(_updateState) Task=3D=
`c887acfa-bd10-4dfb-9374-da607c133e68`::moving from state init -> state =
preparing<br />Thread-55245::INFO::2015-10-12 10:12:22,530::logUtils::44::d=
ispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID=3D'd44ee4b0-8d36=
-467a-9610-c682a618b698', spUUID=3D'0ae7120a-430d-4534-9a7e-59c53fb2e804', =
imgUUID=3D'3454b077-297b-4b89-b8ce-a77f6ec5d22e', volUUID=3D'933da0b6-6a05-=
4e64-958a-e1c030cf5ddb', options=3DNone)<br />Thread-55245::INFO::2015-10-1=
2 10:12:22,535::logUtils::47::dispatcher::(wrapper) Run and protect: getVol=
umeSize, Return response: {'truesize': '158983839744',
'apparentsize': '161=
061273600'}<br />Thread-55245::DEBUG::2015-10-12 10:12:22,535::task::1191::=
Storage.TaskManager.Task::(prepare) Task=3D`c887acfa-bd10-4dfb-9374-da607c1=
33e68`::finished: {'truesize': '158983839744', 'apparentsize':
'16106127360=
0'}<br />Thread-55245::DEBUG::2015-10-12 10:12:22,535::task::595::Storage=
=2ETaskManager.Task::(_updateState) Task=3D`c887acfa-bd10-4dfb-9374-da607c1=
33e68`::moving from state preparing -> state finished<br />Thread-55245:=
:DEBUG::2015-10-12 10:12:22,535::resourceManager::940::Storage.ResourceMana=
ger.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<br />Thre=
ad-55245::DEBUG::2015-10-12 10:12:22,536::resourceManager::977::Storage.Res=
ourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<br />Thread-552=
45::DEBUG::2015-10-12 10:12:22,536::task::993::Storage.TaskManager.Task::(_=
decref) Task=3D`c887acfa-bd10-4dfb-9374-da607c133e68`::ref 0 aborting False=
<br />Thread-55245::DEBUG::2015-10-12 10:12:22,545::libvirtconnection::143:=
:root::(wrapper) Unknown libvirterror: ecode: 80 edom: 20 level: 2 message:=
metadata not found: Requested metadata element is not present<br />JsonRpc=
(StompReactor)::DEBUG::2015-10-12 10:12:23,138::stompReactor::98::Broker=
=2EStompAdapter::(handle_frame) Handling message <StompFrame command=3D'=
SEND'><br />JsonRpcServer::DEBUG::2015-10-12
10:12:23,139::__init__::530=
::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request<br />Thread-1=
67406::DEBUG::2015-10-12 10:12:23,142::stompReactor::163::yajsonrpc.StompSe=
rver::(send) Sending response<br />Thread-37810::DEBUG::2015-10-12 10:12:24=
,194::fileSD::262::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=3D/rh=
ev/data-center/mnt/ovirt01:_data_iso/5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0/d=
om_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 (cwd None)=
<br />Thread-37810::DEBUG::2015-10-12 10:12:24,201::fileSD::262::Storage.Mi=
sc.excCmd::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 rec=
ords out\n317 bytes (317 B) copied, 0.000131729 s, 2.4 MB/s\n'; <rc> =
=3D 0<br />JsonRpc (StompReactor)::DEBUG::2015-10-12 10:12:26,148::stompRea=
ctor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFra=
me command=3D'SEND'><br />JsonRpcServer::DEBUG::2015-10-12
10:12:26,149:=
:__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request=
<br />Thread-167407::DEBUG::2015-10-12 10:12:26,151::stompReactor::163::yaj=
sonrpc.StompServer::(send) Sending response<br />VM Channels Listener::DEBU=
G::2015-10-12 10:12:26,972::vmchannels::96::vds::(_handle_timeouts) Timeout=
on fileno 35.<br />Thread-30::DEBUG::2015-10-12 10:12:28,358::fileSD::262:=
:Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=3D/rhev/data-center/mnt=
/glusterSD/localhost:_ovirt/d44ee4b0-8d36-467a-9610-c682a618b698/dom_md/met=
adata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 (cwd None)<br />Thr=
ead-30::DEBUG::2015-10-12 10:12:28,451::fileSD::262::Storage.Misc.excCmd::(=
getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n47=
0 bytes (470 B) copied, 0.000152738 s, 3.1 MB/s\n'; <rc> =3D 0<br
/>J=
sonRpc (StompReactor)::DEBUG::2015-10-12 10:12:29,157::stompReactor::98::Br=
oker.StompAdapter::(handle_frame) Handling message <StompFrame command=
=3D'SEND'><br />JsonRpcServer::DEBUG::2015-10-12
10:12:29,252::__init__:=
:530::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request<br />Thre=
ad-167408::DEBUG::2015-10-12 10:12:29,254::stompReactor::163::yajsonrpc.Sto=
mpServer::(send) Sending response<br />JsonRpc (StompReactor)::DEBUG::2015-=
10-12 10:12:32,260::stompReactor::98::Broker.StompAdapter::(handle_frame) H=
andling message <StompFrame command=3D'SEND'><br
/>JsonRpcServer::DEB=
UG::2015-10-12 10:12:32,262::__init__::530::jsonrpc.JsonRpcServer::(serve_r=
equests) Waiting for request<br />Thread-167409::DEBUG::2015-10-12 10:12:32=
,264::task::595::Storage.TaskManager.Task::(_updateState) Task=3D`7d55817b-=
a5c4-4c27-b2d5-e892ba645476`::moving from state init -> state preparing<=
br />Thread-167409::INFO::2015-10-12 10:12:32,264::logUtils::44::dispatcher=
::(wrapper) Run and protect: repoStats(options=3DNone)<br />Thread-167409::=
INFO::2015-10-12 10:12:32,265::logUtils::47::dispatcher::(wrapper) Run and =
protect: repoStats, Return response: {u'd44ee4b0-8d36-467a-9610-c682a618b69=
8': {'code': 0, 'actual': True, 'version': 3,
'acquired': True, 'delay': '0=
=2E000152738', 'lastCheck': '3.6', 'valid': True},
u'5aec30fa-be8b-4f4e-832=
e-eafb6fa4a8e0': {'code': 0, 'actual': True, 'version': 0,
'acquired': True=
, 'delay': '0.000131729', 'lastCheck': '8.1',
'valid': True}}<br />Thread-1=
67409::DEBUG::2015-10-12 10:12:32,265::task::1191::Storage.TaskManager.Task=
::(prepare) Task=3D`7d55817b-a5c4-4c27-b2d5-e892ba645476`::finished: {u'd44=
ee4b0-8d36-467a-9610-c682a618b698': {'code': 0, 'actual': True,
'version': =
3, 'acquired': True, 'delay': '0.000152738', 'lastCheck':
'3.6', 'valid': T=
rue}, u'5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0': {'code': 0,
'actual': True, =
'version': 0, 'acquired': True, 'delay': '0.000131729',
'lastCheck': '8.1',=
'valid': True}}<br />Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::=
595::Storage.TaskManager.Task::(_updateState) Task=3D`7d55817b-a5c4-4c27-b2=
d5-e892ba645476`::moving from state preparing -> state finished<br />Thr=
ead-167409::DEBUG::2015-10-12 10:12:32,265::resourceManager::940::Storage=
=2EResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resourc=
es {}<br />Thread-167409::DEBUG::2015-10-12 10:12:32,265::resourceManager::=
977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}=
<br />Thread-167409::DEBUG::2015-10-12 10:12:32,265::task::993::Storage.Tas=
kManager.Task::(_decref) Task=3D`7d55817b-a5c4-4c27-b2d5-e892ba645476`::ref=
0 aborting False<br />Thread-167409::DEBUG::2015-10-12 10:12:32,268::stomp=
Reactor::163::yajsonrpc.StompServer::(send) Sending response<br />JsonRpc (=
StompReactor)::DEBUG::2015-10-12 10:12:32,275::stompReactor::98::Broker.Sto=
mpAdapter::(handle_frame) Handling message <StompFrame
command=3D'SEND'&=
gt;<br />JsonRpcServer::DEBUG::2015-10-12 10:12:32,278::__init__::530::json=
rpc.JsonRpcServer::(serve_requests) Waiting for request<br />Thread-167410:=
:DEBUG::2015-10-12 10:12:32,283::stompReactor::163::yajsonrpc.StompServer::=
(send) Sending response</p <p> </p <div> </div
</body></html
--=_d69a1f5c639decb7303d9c14e20f8797--
On Sun, Oct 11, 2015 at 6:43 PM, Nico <gluster(a)distran.org>
wrote:
> Recently, i built a small oVirt platform with 2 dedicated servers and GlusterFS to
synch the VM storage.
> Bricks:
>
> Brick1: ovirt01:/gluster/ovirt
>
> Brick2: ovirt02:/gluster/ovirt
This looks like replica 2 - this is not supported.
You can use either replica 1 (testing) or replica 3 (production).
> But when i check /var/log/ovirt/engine.log on ovirt01, there are error in loop every
2 seconds:
To understand such error we need to see the vdsm log.
Nir
Yeah it is replica 2 as i've only 2 dedicated servers.
why are you saying it is not supported ? Through oVirt GUI, it is
possible to create a Gluster Volume with 2 bricks in repllcate mode; i
tried it also.
here the last entries of vdsm.log
hread-167405::DEBUG::2015-10-12
10:12:20,132::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
Thread-55245::DEBUG::2015-10-12
10:12:22,529::task::595::Storage.TaskManager.Task::(_updateState)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::moving from state init -
2:04 p.m.
On Mon, Oct 12, 2015 at 11:14 AM, Nico <gluster(a)distran.org> wrote:
Le 2015-10-12 09:59, Nir Soffer a écrit :
On Sun, Oct 11, 2015 at 6:43 PM, Nico <gluster(a)distran.org> wrote:
Recently, i built a small oVirt platform with 2 dedicated servers and
GlusterFS to synch the VM storage.
Bricks:
Brick1: ovirt01:/gluster/ovirt
Brick2: ovirt02:/gluster/ovirt
This looks like replica 2 - this is not supported.
You can use either replica 1 (testing) or replica 3 (production).
But when i check /var/log/ovirt/engine.log on ovirt01, there are error in
loop every 2 seconds:
To understand such error we need to see the vdsm log.
Nir
Yeah it is replica 2 as i've only 2 dedicated servers.
why are you saying it is not supported ? Through oVirt GUI, it is possible
to create a Gluster Volume with 2 bricks in repllcate mode; i tried it also.
Yes, engine will let you use such volume in 3.5 - this is a bug. In 3.6 you will
not be able to use such setup.
replica 2 fails in a very bad way when one brick is down; the
application may get
stale data, and this breaks sanlock. You will be get stuck with spm
that cannot be
stopped and other fun stuff.
You don't want to go in this direction, and we will not be able to support that.
here the last entries of vdsm.log
We need the whole file.
I suggest you file an ovirt bug and attach the full vdsm log file
showing the timeframe of
the error. Probably from the time you created the glusterfs domain.
Nir
hread-167405::DEBUG::2015-10-12
10:12:20,132::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
Thread-55245::DEBUG::2015-10-12
10:12:22,529::task::595::Storage.TaskManager.Task::(_updateState)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::moving from state init -> state
preparing
Thread-55245::INFO::2015-10-12
10:12:22,530::logUtils::44::dispatcher::(wrapper) Run and protect:
getVolumeSize(sdUUID='d44ee4b0-8d36-467a-9610-c682a618b698',
spUUID='0ae7120a-430d-4534-9a7e-59c53fb2e804',
imgUUID='3454b077-297b-4b89-b8ce-a77f6ec5d22e',
volUUID='933da0b6-6a05-4e64-958a-e1c030cf5ddb', options=None)
Thread-55245::INFO::2015-10-12
10:12:22,535::logUtils::47::dispatcher::(wrapper) Run and protect:
getVolumeSize, Return response: {'truesize': '158983839744',
'apparentsize':
'161061273600'}
Thread-55245::DEBUG::2015-10-12
10:12:22,535::task::1191::Storage.TaskManager.Task::(prepare)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::finished: {'truesize':
'158983839744', 'apparentsize': '161061273600'}
Thread-55245::DEBUG::2015-10-12
10:12:22,535::task::595::Storage.TaskManager.Task::(_updateState)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::moving from state preparing ->
state finished
Thread-55245::DEBUG::2015-10-12
10:12:22,535::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-55245::DEBUG::2015-10-12
10:12:22,536::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-55245::DEBUG::2015-10-12
10:12:22,536::task::993::Storage.TaskManager.Task::(_decref)
Task=`c887acfa-bd10-4dfb-9374-da607c133e68`::ref 0 aborting False
Thread-55245::DEBUG::2015-10-12
10:12:22,545::libvirtconnection::143::root::(wrapper) Unknown libvirterror:
ecode: 80 edom: 20 level: 2 message: metadata not found: Requested metadata
element is not present
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:23,138::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling
message <StompFrame command='SEND'>
JsonRpcServer::DEBUG::2015-10-12
10:12:23,139::__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting
for request
Thread-167406::DEBUG::2015-10-12
10:12:23,142::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
Thread-37810::DEBUG::2015-10-12
10:12:24,194::fileSD::262::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd
if=/rhev/data-center/mnt/ovirt01:_data_iso/5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)
Thread-37810::DEBUG::2015-10-12
10:12:24,201::fileSD::262::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
<err> = '0+1 records in\n0+1 records out\n317 bytes (317 B) copied,
0.000131729 s, 2.4 MB/s\n'; <rc> = 0
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:26,148::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling
message <StompFrame command='SEND'>
JsonRpcServer::DEBUG::2015-10-12
10:12:26,149::__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting
for request
Thread-167407::DEBUG::2015-10-12
10:12:26,151::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
VM Channels Listener::DEBUG::2015-10-12
10:12:26,972::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 35.
Thread-30::DEBUG::2015-10-12
10:12:28,358::fileSD::262::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd
if=/rhev/data-center/mnt/glusterSD/localhost:_ovirt/d44ee4b0-8d36-467a-9610-c682a618b698/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)
Thread-30::DEBUG::2015-10-12
10:12:28,451::fileSD::262::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
<err> = '0+1 records in\n0+1 records out\n470 bytes (470 B) copied,
0.000152738 s, 3.1 MB/s\n'; <rc> = 0
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:29,157::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling
message <StompFrame command='SEND'>
JsonRpcServer::DEBUG::2015-10-12
10:12:29,252::__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting
for request
Thread-167408::DEBUG::2015-10-12
10:12:29,254::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:32,260::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling
message <StompFrame command='SEND'>
JsonRpcServer::DEBUG::2015-10-12
10:12:32,262::__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting
for request
Thread-167409::DEBUG::2015-10-12
10:12:32,264::task::595::Storage.TaskManager.Task::(_updateState)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::moving from state init -> state
preparing
Thread-167409::INFO::2015-10-12
10:12:32,264::logUtils::44::dispatcher::(wrapper) Run and protect:
repoStats(options=None)
Thread-167409::INFO::2015-10-12
10:12:32,265::logUtils::47::dispatcher::(wrapper) Run and protect:
repoStats, Return response: {u'd44ee4b0-8d36-467a-9610-c682a618b698':
{'code': 0, 'actual': True, 'version': 3, 'acquired':
True, 'delay':
'0.000152738', 'lastCheck': '3.6', 'valid': True},
u'5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0': {'code': 0, 'actual':
True,
'version': 0, 'acquired': True, 'delay': '0.000131729',
'lastCheck': '8.1',
'valid': True}}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::1191::Storage.TaskManager.Task::(prepare)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::finished:
{u'd44ee4b0-8d36-467a-9610-c682a618b698': {'code': 0, 'actual':
True,
'version': 3, 'acquired': True, 'delay': '0.000152738',
'lastCheck': '3.6',
'valid': True}, u'5aec30fa-be8b-4f4e-832e-eafb6fa4a8e0': {'code':
0,
'actual': True, 'version': 0, 'acquired': True, 'delay':
'0.000131729',
'lastCheck': '8.1', 'valid': True}}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::595::Storage.TaskManager.Task::(_updateState)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::moving from state preparing ->
state finished
Thread-167409::DEBUG::2015-10-12
10:12:32,265::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-167409::DEBUG::2015-10-12
10:12:32,265::task::993::Storage.TaskManager.Task::(_decref)
Task=`7d55817b-a5c4-4c27-b2d5-e892ba645476`::ref 0 aborting False
Thread-167409::DEBUG::2015-10-12
10:12:32,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
JsonRpc (StompReactor)::DEBUG::2015-10-12
10:12:32,275::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling
message <StompFrame command='SEND'>
JsonRpcServer::DEBUG::2015-10-12
10:12:32,278::__init__::530::jsonrpc.JsonRpcServer::(serve_requests) Waiting
for request
Thread-167410::DEBUG::2015-10-12
10:12:32,283::stompReactor::163::yajsonrpc.StompServer::(send) Sending
response
3:13 p.m.
--=_31eaa4a1620c103e9af9f3ded80f2bd2
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
Le 2015-10-12 14:04, Nir Soffer a écrit :
Yes, engine will let you use such volume in 3.5 - this is a bug. In
3.6 you will
not be able to use such setup.
replica 2 fails in a very bad way when one brick is down; the
application may get
stale data, and this breaks sanlock. You will be get stuck with spm
that cannot be
stopped and other fun stuff.
You don't want to go in this direction, and we will not be able to support that.
> here the last entries of vdsm.log
We need the whole file.
I suggest you file an ovirt bug and attach the full vdsm log file
showing the timeframe of
the error. Probably from the time you created the glusterfs domain.
Nir
Please find the full logs there:
https://94.23.2.63/log_vdsm/vdsm.log [1]
https://94.23.2.63/log_vdsm/ [2]
https://94.23.2.63/log_engine/ [3]
Links:
------
[1] https://94.23.2.63/log_vdsm/vdsm.log
[2] https://94.23.2.63/log_vdsm/
[3] https://94.23.2.63/log_engine/
--=_31eaa4a1620c103e9af9f3ded80f2bd2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=
=3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family:
Verdana,Gen=
eva,sans-serif'>
<p>Le 2015-10-12 14:04, Nir Soffer a écrit :</p>
<blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left:
#1010ff 2=
px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!--
meta ig=
nored -->
<div class=3D"pre" style=3D"margin: 0; padding: 0; font-family:
monospace">=
<br /> Yes, engine will let you use such volume in 3.5 - this is a bug. In =
3.6 you will<br /> not be able to use such setup.<br /> <br /> replica 2
fa=
ils in a very bad way when one brick is down; the<br /> application may get=
<br /> stale data, and this breaks sanlock. You will be get stuck with spm<=
br /> that cannot be<br /> stopped and other fun stuff.<br /> <br />
You do=
n't want to go in this direction, and we will not be able to support that=
=2E<br /> <br />
<blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left:
#1010ff 2=
px solid; margin: 0">here the last entries of vdsm.log</blockquote>
<br /> We need the whole file.<br /> <br /> I suggest you file an ovirt
bug=
and attach the full vdsm log file<br /> showing the timeframe of<br /> the=
error. Probably from the time you created the glusterfs domain.<br /> <br =
/> Nir<br /> <br /></div>
</blockquote>
<p> </p>
<p>Please find the full logs there:</p>
<p> </p>
<p><a
href=3D"https://94.23.2.63/log_vdsm/vdsm.log">https://94.23.2.63/log_=
vdsm/vdsm.log</a</p>
<p><a
href=3D"https://94.23.2.63/log_vdsm/">https://94.23.2.63/log_vdsm/</a=
</p>
<p><a
href=3D"https://94.23.2.63/log_engine/">https://94.23.2.63/log_engine=
/</a</p>
<p> </p>
<p> </p>
<p> </p>
<div> </div>
</body></html>
--=_31eaa4a1620c103e9af9f3ded80f2bd2--
4:34 p.m.
This is a multi-part message in MIME format.
--------------090203080600010300040807
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
On 10/12/2015 06:43 PM, Nico wrote:
Le 2015-10-12 14:04, Nir Soffer a écrit :
>
> Yes, engine will let you use such volume in 3.5 - this is a bug. In
> 3.6 you will
> not be able to use such setup.
>
> replica 2 fails in a very bad way when one brick is down; the
> application may get
> stale data, and this breaks sanlock. You will be get stuck with spm
> that cannot be
> stopped and other fun stuff.
>
> You don't want to go in this direction, and we will not be able to
> support that.
>
>> here the last entries of vdsm.log
>
> We need the whole file.
>
> I suggest you file an ovirt bug and attach the full vdsm log file
> showing the timeframe of
> the error. Probably from the time you created the glusterfs domain.
>
> Nir
>
Please find the full logs there:
https://94.23.2.63/log_vdsm/vdsm.log
https://94.23.2.63/log_vdsm/
https://94.23.2.63/log_engine/
The engine log looping with "Volume contains apparently corrupt bricks"-
is when engine tries to get information from gluster CLI about the
volumes and updates its database. These errors do not affect the
functioning of the storage domain and running virtual machines, but
affect the monitoring/management of the gluster volume from oVirt.
Now to identify the cause of the error - the logs indicate that the
gluster's server uuid has either not been updated/ or is different in
the engine. Could be one of these scenarios
1. Did you create the cluster with only virt service enabled and later
enable gluster service? In this case, the gluster server uuid may not be
updated. You will need to put host to maintenance and then activate it
to resolve this
2. Did you re-install the gluster server nodes after adding it to oVirt?
If this is the case, we need to investigate further how there's a mismatch.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------090203080600010300040807
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 10/12/2015 06:43 PM, Nico
wrote:<br>
</div>
<blockquote cite="mid:90c06d56781006f10ad99b5ad02fe43a@lienard.name"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<p>Le 2015-10-12 14:04, Nir Soffer a écrit :</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head
ignored --><!-- meta ignored -->
<div class="pre" style="margin: 0; padding: 0; font-family:
monospace"><br>
Yes, engine will let you use such volume in 3.5 - this is a
bug. In 3.6 you will<br>
not be able to use such setup.<br>
<br>
replica 2 fails in a very bad way when one brick is down; the<br>
application may get<br>
stale data, and this breaks sanlock. You will be get stuck
with spm<br>
that cannot be<br>
stopped and other fun stuff.<br>
<br>
You don't want to go in this direction, and we will not be
able to support that.<br>
<br>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0">here the last entries of
vdsm.log</blockquote>
<br>
We need the whole file.<br>
<br>
I suggest you file an ovirt bug and attach the full vdsm log
file<br>
showing the timeframe of<br>
the error. Probably from the time you created the glusterfs
domain.<br>
<br>
Nir<br>
<br>
</div>
</blockquote>
<p> </p>
<p>Please find the full logs there:</p>
<p> </p>
<p><a moz-do-not-send="true"
href="https://94.23.2.63/log_vdsm/vdsm.log">https://94.23.2.63/log_vdsm/vdsm.log</a></p>
<p><a moz-do-not-send="true"
href="https://94.23.2.63/log_vdsm/">https://94.23.2.63/log_vdsm/</a></p>
<p><a moz-do-not-send="true"
href="https://94.23.2.63/log_engine/">https://94.23.2.63/log_engine/</a></p>
</blockquote>
<br>
The engine log looping with "Volume contains apparently corrupt
bricks"- is when engine tries to get information from gluster CLI
about the volumes and updates its database. These errors do not
affect the functioning of the storage domain and running virtual
machines, but affect the monitoring/management of the gluster volume
from oVirt.<br>
<br>
Now to identify the cause of the error - the logs indicate that the
gluster's server uuid has either not been updated/ or is different
in the engine. Could be one of these scenarios<br>
1. Did you create the cluster with only virt service enabled and
later enable gluster service? In this case, the gluster server uuid
may not be updated. You will need to put host to maintenance and
then activate it to resolve this<br>
<br>
2. Did you re-install the gluster server nodes after adding it to
oVirt? If this is the case, we need to investigate further how
there's a mismatch.<br>
<br>
<br>
<blockquote cite="mid:90c06d56781006f10ad99b5ad02fe43a@lienard.name"
type="cite">
<p> </p>
<p> </p>
<p> </p>
<div> </div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------090203080600010300040807--
4:08 p.m.
--=_22a8f6a92661e1f6532f09f6ce07eee9
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
Le 2015-10-12 14:04, Nir Soffer a écrit :
> On Mon, Oct 12, 2015 at 11:14 AM, Nico
<gluster(a)distran.org> wrote:
Yes, engine will let you use such volume in 3.5 - this is a bug. In 3.6 you will
not be able to use such setup.
replica 2 fails in a very bad way when one brick is down; the
application may get
stale data, and this breaks sanlock. You will be get stuck with spm
that cannot be
stopped and other fun stuff.
You don't want to go in this direction, and we will not be able to support that.
For the record, I already rebooted node1; and the node2 took over the
existing VM from node 1 and vice-versa.
GlusterFS worked fine, oVirt application was still working fine .. i
guess it is because it was a soft reboot which stops softly the
services.
I got another case where i stuck the network on the 2 nodes
simultaneously after a bad manipulation on oVirt GUI and i got a split
brain.
i kept the error at this very moment:
root@devnix-virt-master02 nets]# gluster volume heal ovirt info
split-brain
Brick devnix-virt-master01:/gluster/ovirt/
/d44ee4b0-8d36-467a-9610-c682a618b698/dom_md/ids
Number of entries in split-brain: 1
Brick devnix-virt-master02:/gluster/ovirt/
/d44ee4b0-8d36-467a-9610-c682a618b698/dom_md/ids
Number of entries in split-brain: 1
This file was having same size on both nodes; so it was hard to select
one. Finally i chose the younger one and all was back online after the
heal.
It is this kind of stuff you are talking about with 2 nodes ?
For now, I don't have budget to take a third one; so i'm a bit stuck and
disappointing.
I've a third device but for backup, it has lot of storage but low cpu
abilities (no VT-X) so i can't use it as hypervisor.
I could maybe use it as a third brick but is it possible to have this
kind of configuration ? 2 actives nodes as hypervisor and 1 third only
for gluster replica 3 ?
Cheers
Nico
--=_22a8f6a92661e1f6532f09f6ce07eee9
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=
=3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family:
Verdana,Gen=
eva,sans-serif'>
<p>Le 2015-10-12 14:04, Nir Soffer a écrit :</p>
<blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left:
#1010ff 2=
px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!--
meta ig=
nored -->
<div class=3D"pre" style=3D"margin: 0; padding: 0; font-family:
monospace">
<blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left:
#1010ff 2=
px solid; margin: 0">On Mon, Oct 12, 2015 at 11:14 AM, Nico <<a
href=3D"=
mailto:gluster@distran.org">gluster@distran.org</a>>
wrote:</blockquote>
<br /> Yes, engine will let you use such volume in 3.5 - this is a bug. In =
3.6 you will<br /> not be able to use such setup.<br /> <br /> replica 2
fa=
ils in a very bad way when one brick is down; the<br /> application may get=
<br /> stale data, and this breaks sanlock. You will be get stuck with spm<=
br /> that cannot be<br /> stopped and other fun stuff.<br /> <br />
You do=
n't want to go in this direction, and we will not be able to support that=
=2E<br /> </div>
</blockquote>
<p> </p>
<p>For the record, I already rebooted node1; and the node2 took over the ex=
isting VM from node 1 and vice-versa.</p>
<p>GlusterFS worked fine, oVirt application was still working
fin=
e .. i guess it is because it was a soft reboot which stops soft=
ly the services.</p>
<p>I got another case where i stuck the network on the 2 nodes simultaneous=
ly after a bad manipulation on oVirt GUI and i got a split brain.</p>
<p>i kept the error at this very moment:</p>
<p>root@devnix-virt-master02 nets]# gluster volume heal ovirt info split-br=
ain<br />Brick devnix-virt-master01:/gluster/ovirt/<br />/d44ee4b0-8d36-467=
a-9610-c682a618b698/dom_md/ids<br />Number of entries in split-brain: 1</p>
<p>Brick devnix-virt-master02:/gluster/ovirt/<br />/d44ee4b0-8d36-467a-9610=
-c682a618b698/dom_md/ids<br />Number of entries in split-brain: 1</p>
<p> </p>
<p> </p>
<p>This file was having same size on both nodes; so it was hard to select o=
ne. Finally i chose the younger one and all was back online after the heal=
=2E</p>
<p>It is this kind of stuff you are talking about with 2 nodes ?</p>
<p> </p>
<p>For now, I don't have budget to take a third one; so i'm a bit stuck
and=
disappointing.</p>
<p>I've a third device but for backup, it has lot of storage but low cpu ab=
ilities (no VT-X) so i can't use it as hypervisor.</p>
<p>I could maybe use it as a third brick but is it possible to have this ki=
nd of configuration ? 2 actives nodes as hypervisor and 1 third only for gl=
uster replica 3 ?</p>
<p>Cheers</p>
<p>Nico</p>
<p> </p>
<p> </p>
<div> </div>
</body></html>
--=_22a8f6a92661e1f6532f09f6ce07eee9--
3139
days inactive
3141
days old
6 comments
3 participants
participants (3)
-
Nico -
Nir Soffer -
Sahina Bose