<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I hadn’t revisited it yet, but it is possible to use cgroups to limit glusterfs’s cpu usage, might help you out.<div class=""><br class=""></div><div class="">Andrew Wklau has a blog post about it: <a href="http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/" class="">http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/</a></div><div class=""><br class=""></div><div class="">Careful about how far you throttle it down, if it’s your VMs disk it’s rebuilding, you’ll pause it anyway I’d expect.</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 4, 2015, at 8:57 AM, Jorick Astrego <<a href="mailto:j.astrego@netbulae.eu" class="">j.astrego@netbulae.eu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/03/2015 10:04 PM, Alastair Neil
wrote:<br class="">
</div>
<blockquote cite="mid:CA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w@mail.gmail.com" type="cite" class="">
<div dir="ltr" class="">Any follow up on this?
<div class=""><br class="">
</div>
<div class=""> Are there known issues using a replica 3 glsuter datastore
with lvm thin provisioned bricks?</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 20 March 2015 at 15:22, Alastair
Neil <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:ajneil.tech@gmail.com" target="_blank" class="">ajneil.tech@gmail.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">CentOS 6.6</div>
<span class="">
<div class=""> </div>
<blockquote class="gmail_quote" style="font-size:13px;margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> vdsm-4.16.10-8.gitc937927.el6<br class="">
glusterfs-3.6.2-1.el6<br class="">
2.6.32 - 504.8.1.el6.x86_64</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">moved to 3.6 specifically to get the snapshotting
feature, hence my desire to migrate to thinly
provisioned lvm bricks.</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
<br class="">
Well on the glusterfs mailinglist there have been discussions:<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">3.6.2 is a major release and introduces some
new features in cluster wide concept. Additionally it is not
stable yet.</blockquote>
<br class="">
<br class="">
<br class="">
<br class="">
<blockquote cite="mid:CA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w@mail.gmail.com" type="cite" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 20 March 2015 at 14:57,
Darrell Budic <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:budic@onholyground.com" target="_blank" class="">budic@onholyground.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">What version of
gluster are you running on these?
<div class=""><br class="">
</div>
<div class="">I’ve seen high load during heals bounce my
hosted engine around due to overall system load,
but never pause anything else. Cent 7 combo
storage/host systems, gluster 3.5.2.</div>
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Mar 20, 2015, at 9:57 AM,
Alastair Neil <<a moz-do-not-send="true" href="mailto:ajneil.tech@gmail.com" target="_blank" class="">ajneil.tech@gmail.com</a>>
wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Pranith
<div class=""><br class="">
</div>
<div class="">I have run a pretty
straightforward test. I created a
two brick 50 G replica volume with
normal lvm bricks, and installed
two servers, one centos 6.6 and
one centos 7.0. I kicked off
bonnie++ on both to generate some
file system activity and then made
the volume replica 3. I saw no
issues on the servers. </div>
<div class=""><br class="">
</div>
<div class="">Not clear if this is a
sufficiently rigorous test and the
Volume I have had issues on is a
3TB volume with about 2TB used.</div>
<div class=""><br class="">
</div>
<div class="">-Alastair</div>
<div class=""><br class="">
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 19
March 2015 at 12:30, Alastair
Neil <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:ajneil.tech@gmail.com" target="_blank" class="">ajneil.tech@gmail.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr" class="">I don't think I
have the resources to test
it meaningfully. I have
about 50 vms on my primary
storage domain. I might be
able to set up a small 50 GB
volume and provision 2 or 3
vms running test loads but
I'm not sure it would be
comparable. I'll give it a
try and let you know if I
see similar behaviour.</div>
<div class="">
<div class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On
19 March 2015 at
11:34, Pranith Kumar
Karampuri <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" target="_blank" class="">pkarampu@redhat.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF" class="">
Without thinly
provisioned lvm.<span class=""><font color="#888888" class=""><br class="">
<br class="">
Pranith</font></span>
<div class="">
<div class=""><br class="">
<div class="">On
03/19/2015
08:01 PM,
Alastair Neil
wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">do
you mean raw
partitions as
bricks or
simply with
out thin
provisioned
lvm?
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On
19 March 2015
at 00:32,
Pranith Kumar
Karampuri <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" target="_blank" class="">pkarampu@redhat.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF" class=""> Could you let me know if you see this problem without
lvm as well?<span class=""><font color="#888888" class=""><br class="">
<br class="">
Pranith</font></span>
<div class="">
<div class=""><br class="">
<div class="">On
03/18/2015
08:25 PM,
Alastair Neil
wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">I
am in the
process of
replacing the
bricks with
thinly
provisioned
lvs yes.
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On
18 March 2015
at 09:35,
Pranith Kumar
Karampuri <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" target="_blank" class="">pkarampu@redhat.com</a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF" class=""> hi,<br class="">
Are you
using thin-lvm
based backend
on which the
bricks are
created?<br class="">
<br class="">
Pranith
<div class="">
<div class=""><br class="">
<div class="">On
03/18/2015
02:05 AM,
Alastair Neil
wrote:<br class="">
</div>
</div>
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">
<div dir="ltr" class="">I
have a Ovirt
cluster with 6
VM hosts and 4
gluster nodes.
There are two
virtualisation
clusters one
with two
nehelem nodes
and one with
four
sandybridge
nodes. My
master storage
domain is a
GlusterFS
backed by a
replica 3
gluster volume
from 3 of the
gluster
nodes. The
engine is a
hosted engine
3.5.1 on 3 of
the
sandybridge
nodes, with
storage
broviede by
nfs from a
different
gluster
volume. All
the hosts are
CentOS 6.6.
<div class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> vdsm-4.16.10-8.gitc937927.el6<br class="">
glusterfs-3.6.2-1.el6<br class="">
2.6.32 -
504.8.1.el6.x86_64</blockquote>
<div class=""><br class="">
</div>
<div class="">Problems
happen when I
try to add a
new brick or
replace a
brick
eventually the
self heal will
kill the VMs.
In the VM's
logs I see
kernel hung
task
messages. </div>
<div class=""><br class="">
</div>
<div class="">
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel: INFO:
task
nginx:1736
blocked for
more than 120
seconds.<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
Not tainted
2.6.32-504.3.3.el6.x86_64
#1<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel: "echo
0 >
/proc/sys/kernel/hung_task_timeout_secs"
disables this
message.<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel: nginx
D
0000000000000001
0 1736
1735
0x00000080<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
ffff8800778b17a8
0000000000000082
0000000000000000
00000000000126c0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
ffff88007e5c6500
ffff880037170080
0006ce5c85bd9185
ffff88007e5c64d0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
ffff88007a614ae0
00000001722b64ba
ffff88007a615098
ffff8800778b1fd8<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel: Call
Trace:<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8152a885>]
schedule_timeout+0x215/0x2e0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8152a503>]
wait_for_common+0x123/0x180<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff81064b90>]
?
default_wake_function+0x0/0x20<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa0210a76>]
?
_xfs_buf_read+0x46/0x60
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa02063c7>]
?
xfs_trans_read_buf+0x197/0x410
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8152a61d>]
wait_for_completion+0x1d/0x20<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa020ff5b>]
xfs_buf_iowait+0x9b/0x100
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa02063c7>]
?
xfs_trans_read_buf+0x197/0x410
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa0210a76>]
_xfs_buf_read+0x46/0x60
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa0210b3b>]
xfs_buf_read+0xab/0x100
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa02063c7>]
xfs_trans_read_buf+0x197/0x410
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa01ee6a4>]
xfs_imap_to_bp+0x54/0x130
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa01f077b>]
xfs_iread+0x7b/0x1b0
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff811ab77e>]
?
inode_init_always+0x11e/0x1c0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa01eb5ee>]
xfs_iget+0x27e/0x6e0
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa01eae1d>]
?
xfs_iunlock+0x5d/0xd0
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa0209366>]
xfs_lookup+0xc6/0x110
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa0216024>]
xfs_vn_lookup+0x54/0xa0
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8119dc65>]
do_lookup+0x1a5/0x230<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8119e8f4>]
__link_path_walk+0x7a4/0x1000<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff811738e7>]
?
cache_grow+0x217/0x320<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8119f40a>]
path_walk+0x6a/0xe0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8119f61b>]
filename_lookup+0x6b/0xc0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff811a0747>]
user_path_at+0x57/0xa0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa0204e74>]
?
_xfs_trans_commit+0x214/0x2a0
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffffa01eae3e>]
?
xfs_iunlock+0x7e/0xd0
[xfs]<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff81193bc0>]
vfs_fstatat+0x50/0xa0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff811aaf5d>]
?
touch_atime+0x14d/0x1a0<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff81193d3b>]
vfs_stat+0x1b/0x20<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff81193d64>]
sys_newstat+0x24/0x50<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff810e5c87>]
?
audit_syscall_entry+0x1d7/0x200<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff810e5a7e>]
?
__audit_syscall_exit+0x25e/0x290<br class="">
</font><font face="monospace,
monospace" class="">Mar
12 23:05:16
static1
kernel:
[<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b</font></blockquote>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I am
wondering if
my volume
settings are
causing this.
Can anyone
with more
knowledge take
a look and let
me know:</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><font face="monospace,
monospace" class="">network.remote-dio:
on<br class="">
</font><font face="monospace,
monospace" class="">performance.stat-prefetch:
off<br class="">
</font><font face="monospace,
monospace" class="">performance.io-cache:
off<br class="">
</font><font face="monospace,
monospace" class="">performance.read-ahead:
off<br class="">
</font><font face="monospace,
monospace" class="">performance.quick-read:
off<br class="">
</font><font face="monospace,
monospace" class="">nfs.export-volumes:
on<br class="">
</font><font face="monospace,
monospace" class="">network.ping-timeout:
20<br class="">
</font><font face="monospace,
monospace" class="">cluster.self-heal-readdir-size:
64KB<br class="">
</font><font face="monospace,
monospace" class="">cluster.quorum-type:
auto<br class="">
</font><font face="monospace,
monospace" class="">cluster.data-self-heal-algorithm:
diff<br class="">
</font><font face="monospace,
monospace" class="">cluster.self-heal-window-size:
8<br class="">
</font><font face="monospace,
monospace" class="">cluster.heal-timeout:
500<br class="">
</font><font face="monospace,
monospace" class="">cluster.self-heal-daemon:
on<br class="">
</font><font face="monospace,
monospace" class="">cluster.entry-self-heal:
on<br class="">
</font><font face="monospace,
monospace" class="">cluster.data-self-heal:
on<br class="">
</font><font face="monospace,
monospace" class="">cluster.metadata-self-heal:
on<br class="">
</font><font face="monospace,
monospace" class="">cluster.readdir-optimize:
on<br class="">
</font><font face="monospace,
monospace" class="">cluster.background-self-heal-count:
20<br class="">
</font><font face="monospace,
monospace" class="">cluster.rebalance-stats:
on<br class="">
</font><font face="monospace,
monospace" class="">cluster.min-free-disk:
5%<br class="">
</font><font face="monospace,
monospace" class="">cluster.eager-lock:
enable<br class="">
</font><font face="monospace,
monospace" class="">storage.owner-uid:
36<br class="">
</font><font face="monospace,
monospace" class="">storage.owner-gid:
36<br class="">
</font><font face="monospace,
monospace" class="">auth.allow:*<br class="">
</font><font face="monospace,
monospace" class="">user.cifs:
disable<br class="">
</font><font face="monospace,
monospace" class="">cluster.server-quorum-ratio:
51%</font></blockquote>
</div>
<div class=""><br class="">
</div>
<div class="">Many
Thanks,
Alastair</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<fieldset class=""></fieldset>
<br class="">
</div>
</div>
<pre class="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank" class="">Users@ovirt.org</a>
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank" class="">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br class="">
</div>
<br class="">
_______________________________________________<br class="">
Users mailing
list<br class="">
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank" class="">Users@ovirt.org</a><br class="">
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank" class="">http://lists.ovirt.org/mailman/listinfo/users</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
Users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank" class="">Users@ovirt.org</a><br class="">
<a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank" class="">http://lists.ovirt.org/mailman/listinfo/users</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<div class="HOEnZb">
<div class="h5"><br class="">
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class="">
<br class="">
<blockquote cite="mid:CA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w@mail.gmail.com" type="cite" class="">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb">
<div class="h5">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br class="">
<br class="">
<br class="">
<b style="color:#604c78" class=""></b><br class=""><span style="color:#604c78;" class=""><span style="mso-fareast-language:en-gb;" lang="NL" class="">Met vriendelijke groet, With kind regards,<br class=""><br class=""></span>Jorick Astrego</span><b style="color:#604c78" class=""><br class=""><br class="">Netbulae Virtualization Experts </b><br class=""><hr style="border:none;border-top:1px solid #ccc;" class=""><table style="width: 522px" class=""><tbody class=""><tr class=""><td style="width: 130px;font-size: 10px" class="">Tel: 053 20 30 270</td> <td style="width: 130px;font-size: 10px" class=""><a href="mailto:info@netbulae.eu" class="">info@netbulae.eu</a></td> <td style="width: 130px;font-size: 10px" class="">Staalsteden 4-3A</td> <td style="width: 130px;font-size: 10px" class="">KvK 08198180</td></tr><tr class=""> <td style="width: 130px;font-size: 10px" class="">Fax: 053 20 30 271</td> <td style="width: 130px;font-size: 10px" class=""><a href="http://www.netbulae.eu" class="">www.netbulae.eu</a></td> <td style="width: 130px;font-size: 10px" class="">7547 TA Enschede</td> <td style="width: 130px;font-size: 10px" class="">BTW NL821234584B01</td></tr></tbody></table><br class=""><hr style="border:none;border-top:1px solid #ccc;" class=""><br class="">
</div>
_______________________________________________<br class="">Users mailing list<br class=""><a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/users<br class=""></div></blockquote></div><br class=""></div></body></html>