9:01 p.m.
It appears that VDSM changes the following params:
vm.dirty_ratio = 5
vm.dirty_background_ratio = 2
Any idea why? Because we use cache=none it's irrelevant anyway?
TIA,
Y.
8:48 a.m.
On 29/11/16 22:01 +0200, Yaniv Kaul wrote:
It appears that VDSM changes the following params:
vm.dirty_ratio = 5
vm.dirty_background_ratio = 2
Any idea why? Because we use cache=none it's irrelevant anyway?
It's not really irrelevant, the host still uses disk cache. Anyway,
there is BZ[1] with a presentation[2] that (imho reasonably) states:
"Reduce dirty page limits in KVM host to allow
direct I/O writer VMs to compete successfully
with buffered writer processes for storage
access"
I wonder why virtual-host tuned profile doesn't contain these values:
$ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
vm.dirty_background_ratio = 5
[1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
TIA,
Y.
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
9:13 a.m.
On Wed, Nov 30, 2016 at 9:48 AM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
On 29/11/16 22:01 +0200, Yaniv Kaul wrote:
> It appears that VDSM changes the following params:
> vm.dirty_ratio = 5
> vm.dirty_background_ratio = 2
>
> Any idea why? Because we use cache=none it's irrelevant anyway?
>
It's not really irrelevant, the host still uses disk cache. Anyway,
there is BZ[1] with a presentation[2] that (imho reasonably) states:
"Reduce dirty page limits in KVM host to allow
direct I/O writer VMs to compete successfully
with buffered writer processes for storage
access"
Thanks, but it really makes no sense to me. The direct IO by the VMs is
going to a different storage than what the host is writing to, in most
cases.
The host would write to the local disk, the VMs - to a shared storage,
across NFS or block layer or so. Moreover, their IO is not buffered.
There is very little IO coming from the host itself, generally (I hope
so!).
Partially unrelated - the trend today is actually to put NOOP on the VMs -
the deadline is quite meaningless, as the host scheduler will reschedule
anyway as it see fits.
Most likely it is also a deadline scheduler (but could be NOOP as well if
it's an all flash array, for example). Therefore there is no reason for
anything but simple NOOP on the VMs themselves.
In short, I think it's an outdated decision that perhaps should be
revisited. Not urgent, though.
Y.
I wonder why virtual-host tuned profile doesn't contain these
values:
$ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
vm.dirty_background_ratio = 5
[1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/
rhev-vm-rsptime.pdf
TIA,
> Y.
>
_______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
1:11 p.m.
--kno3LaHbtea9cJj5CQNI45l6V8XacJLPD
Content-Type: multipart/mixed; boundary="xpKKJkTBj1DGO23WMiqFancuNdxdtwPOt";
protected-headers="v1"
From: Sven Kieske <s.kieske(a)mittwald.de>
To: devel(a)ovirt.org
Message-ID: <11712170-1758-96b5-adb8-eb29d59be1b8(a)mittwald.de>
Subject: Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?
References: <CAJgorsbMk1UeLt82GOWsVKOmj=Kse+4CvhoT2E_kj=PUnhG28A(a)mail.gmail.com>
<20161130074847.GA1215(a)dhcp130-218.brq.redhat.com>
<CAJgorsYqRX84st-jByqVbaUHCX=TmtnQeXw1=6FPu_Mbz6kUzg(a)mail.gmail.com>
In-Reply-To: <CAJgorsYqRX84st-jByqVbaUHCX=TmtnQeXw1=6FPu_Mbz6kUzg(a)mail.gmail.com>
--xpKKJkTBj1DGO23WMiqFancuNdxdtwPOt
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 30/11/16 09:13, Yaniv Kaul wrote:
Thanks, but it really makes no sense to me. The direct IO by the VMs
is=
going to a different storage than what the host is writing to, in
most
cases.
The host would write to the local disk, the VMs - to a shared storage,
across NFS or block layer or so. Moreover, their IO is not buffered.
There is very little IO coming from the host itself, generally (I hope
so!).
=20
Partially unrelated - the trend today is actually to put NOOP on the VM=
s -
the deadline is quite meaningless, as the host scheduler will
reschedul=
e
anyway as it see fits.
Most likely it is also a deadline scheduler (but could be NOOP as well =
if
it's an all flash array, for example). Therefore there is no
reason for=
anything but simple NOOP on the VMs themselves.
=20
In short, I think it's an outdated decision that perhaps should be
revisited. Not urgent, though.
Y.
I know it's not the most cared about usecase
but I'd like to add that this might affect local storage
domains, which I happen to use a lot, and maybe others too.
--=20
Mit freundlichen Gr=FC=DFen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
K=F6nigsberger Stra=DFe 6
32339 Espelkamp
T: +495772 293100
F: +495772 293333
https://www.mittwald.de
Gesch=E4ftsf=FChrer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhause=
n
Komplement=E4rin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynha=
usen
--xpKKJkTBj1DGO23WMiqFancuNdxdtwPOt--
--kno3LaHbtea9cJj5CQNI45l6V8XacJLPD
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBAgAGBQJYPsIPAAoJEMby9TMDAbQRCwgQANGx3ttkYz9kbs6x4+vUKiwZ
wD3OgXGeqQVoVIJ7H98JgoK3MRbR6EiPJYrx8A/2ROa1k3JCY96wcl1oJwCWQkl6
AXmU/fgqIXCx/8+0CiDU3VtZtlUVHN4Tr0sVpPJ/mH9YdcY6kGj6NA8b9FgeS7IZ
DxYDXNyAFLCg6Ei3IFsg0UgIvEZz4ckdDLAVwnWBqIVybY62W96tAH8aghCLNZxz
/QnVaW7wOXhZTcMUePGShP2Zy/vW9+P8hzar2ZX4G0Qz86B3lJ6kcFPNuR2D3AIR
U2Mx0NRwnCfQXL2Iv1g4LSecwoUZ8qTqnmBE/qjVeGe7kIFD+RfQABPH9+Yj0Fo9
+28rgx049FLI8DPAHPVyxkd2ClLWZ4XpBC0uDvqvzZcFdJ6URAKu+r3y+IM0q6BG
CI82Ofv50oXbh1zkWRtzHm/A4tHcIO47WcF55GSgyrdWg1Js7UcFzTuSj9weONVi
Af1jsUvjMl1nK98xX8dSXA8qpkAC8V9n/jRd9dIn0bEUkAvQtkkdOzafWkhhvh6r
Esj0iKnzGGilx3+wL6AEd/BexIPwp5QrtUHb7EBT7dwOsKfE2mQt2U2WmWE9WWQ4
+eOCYXXIV/za6cUHy/HxQWHv/ByIuGb7BeX1kEaaWjWmfF2JIfWaZnFgi2MD8u/I
dgNzTJScMAj6cCK0a1vm
=x/5A
-----END PGP SIGNATURE-----
--kno3LaHbtea9cJj5CQNI45l6V8XacJLPD--
1:16 p.m.
On 30/11/16 13:11 +0100, Sven Kieske wrote:
On 30/11/16 09:13, Yaniv Kaul wrote:
> Thanks, but it really makes no sense to me. The direct IO by the VMs is
> going to a different storage than what the host is writing to, in most
> cases.
> The host would write to the local disk, the VMs - to a shared storage,
> across NFS or block layer or so. Moreover, their IO is not buffered.
> There is very little IO coming from the host itself, generally (I hope
> so!).
>
> Partially unrelated - the trend today is actually to put NOOP on the VMs -
> the deadline is quite meaningless, as the host scheduler will reschedule
> anyway as it see fits.
> Most likely it is also a deadline scheduler (but could be NOOP as well if
> it's an all flash array, for example). Therefore there is no reason for
> anything but simple NOOP on the VMs themselves.
>
> In short, I think it's an outdated decision that perhaps should be
> revisited. Not urgent, though.
> Y.
I know it's not the most cared about usecase
but I'd like to add that this might affect local storage
domains, which I happen to use a lot, and maybe others too.
Current values seem to be optimal; considering this use case I'd
definitely leave it in place.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +495772 293100
F: +495772 293333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2:26 p.m.
On Wed, Nov 30, 2016 at 2:11 PM, Sven Kieske <s.kieske(a)mittwald.de> wrote:
On 30/11/16 09:13, Yaniv Kaul wrote:
> Thanks, but it really makes no sense to me. The direct IO by the VMs is
> going to a different storage than what the host is writing to, in most
> cases.
> The host would write to the local disk, the VMs - to a shared storage,
> across NFS or block layer or so. Moreover, their IO is not buffered.
> There is very little IO coming from the host itself, generally (I hope
> so!).
>
> Partially unrelated - the trend today is actually to put NOOP on the VMs
-
> the deadline is quite meaningless, as the host scheduler will reschedule
> anyway as it see fits.
> Most likely it is also a deadline scheduler (but could be NOOP as well if
> it's an all flash array, for example). Therefore there is no reason for
> anything but simple NOOP on the VMs themselves.
>
> In short, I think it's an outdated decision that perhaps should be
> revisited. Not urgent, though.
> Y.
I know it's not the most cared about usecase
but I'd like to add that this might affect local storage
domains, which I happen to use a lot, and maybe others too.
In which case we should change the default for local storage.
BTW, when using Gluster, they have returned these values to the original
values...
Y.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +495772 293100
F: +495772 293333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
1:10 p.m.
--jqM3pA5AEqqKPdR1mpM0Heahc81JGL6Ve
Content-Type: multipart/mixed; boundary="JmHWW6MUOGuane1dNFbNJhlcrxhH1obJ5";
protected-headers="v1"
From: Sven Kieske <s.kieske(a)mittwald.de>
To: devel(a)ovirt.org
Message-ID: <918b520b-176c-3a09-d7c9-a16340ac6480(a)mittwald.de>
Subject: Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?
References: <CAJgorsbMk1UeLt82GOWsVKOmj=Kse+4CvhoT2E_kj=PUnhG28A(a)mail.gmail.com>
<20161130074847.GA1215(a)dhcp130-218.brq.redhat.com>
In-Reply-To: <20161130074847.GA1215(a)dhcp130-218.brq.redhat.com>
--JmHWW6MUOGuane1dNFbNJhlcrxhH1obJ5
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 30/11/16 08:48, Martin Polednik wrote:
It's not really irrelevant, the host still uses disk cache.
Anyway,
there is BZ[1] with a presentation[2] that (imho reasonably) states:
=20
"Reduce dirty page limits in KVM host to allow
direct I/O writer VMs to compete successfully
with buffered writer processes for storage
access"
=20
I wonder why virtual-host tuned profile doesn't contain these values:
=20
$ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
vm.dirty_background_ratio =3D 5
=20
[1]https://bugzilla.redhat.com/show_bug.cgi?id=3D740887
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime=
=2Epdf
Could you share [2] with the wider community? This would be awesome!
--=20
Mit freundlichen Gr=FC=DFen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
K=F6nigsberger Stra=DFe 6
32339 Espelkamp
T: +495772 293100
F: +495772 293333
https://www.mittwald.de
Gesch=E4ftsf=FChrer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhause=
n
Komplement=E4rin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynha=
usen
--JmHWW6MUOGuane1dNFbNJhlcrxhH1obJ5--
--jqM3pA5AEqqKPdR1mpM0Heahc81JGL6Ve
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBAgAGBQJYPsG3AAoJEMby9TMDAbQRIgIQAJAoKRFa2S8zBCIPKDQQz8GL
yFI7jrbveoxYoSzvUU/E+bIEaY1ZvltLjcGaVrqaRG9MAYKNqdGwRzIWqHg7/9dJ
io2E5MO1sIUdwizMozuKobGspW4vpFVuWPpJfZdUt2fOQmZw0lEG8t3loUX9Mzu7
n96q+Zx52y32MrwDqkOZLXQsp9nEmySef0maz+36RWp/vyYyN3Yx1+WZcZe5CqSs
UlnudfGczPVgu+lSi8utFw+C5sKGNaGn4TCswWpkihEvRiQtULJS5drEZRI3vJJP
pqoWBVtPsbZFiGCL34LWpm8oawKI/7xYLwUY4X6YJ3WLwqM2fy6P95rIgR16nMoq
GHkMLhkYb3IkfBquNVpJKc1mT7PAEtKyrbnSJ4mYLSbUIJ3aZZXIHRgTMJ1jcfHG
P4f+khPmMieKeTc/tcWwUo8GEcF/v15se24iOXZKqzzR9H10Hk4jjLzFxhVDgamR
1RW2HoYrs6kyH+K4JwwsAtLX/Y7Kynu5A/q71FzlFjWuSVjjQ/TwWw88Hlplw9Ha
6FNlgzMFVz+L/AbxczrwDmLxphfYGHFxj8fbU7D7CVyJcdPsaRFCE57Ll1mCQm/g
xAyPuRqqao4a16UmNvSrLYWX5unwsgF0KBTvKK49ktniis7wYbHgqbBIvm2tKY3j
utQ+pz/rvDYLPskThwBn
=jyJe
-----END PGP SIGNATURE-----
--jqM3pA5AEqqKPdR1mpM0Heahc81JGL6Ve--
1:22 p.m.
On 30/11/16 13:10 +0100, Sven Kieske wrote:
On 30/11/16 08:48, Martin Polednik wrote:
> It's not really irrelevant, the host still uses disk cache. Anyway,
> there is BZ[1] with a presentation[2] that (imho reasonably) states:
>
> "Reduce dirty page limits in KVM host to allow
> direct I/O writer VMs to compete successfully
> with buffered writer processes for storage
> access"
>
> I wonder why virtual-host tuned profile doesn't contain these values:
>
> $ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
> vm.dirty_background_ratio = 5
>
> [1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
Could you share [2] with the wider community? This would be awesome!
Sorry, I've totally missed the fact it's internal link (unfortunately
publicly visible on the BZ). I believe it's slightly outdated, but
let's ask the author.
Ben, is the document[2] somehow still valid and could it be made publicly
available?
Re-referencing for completeness:
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +495772 293100
F: +495772 293333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:38 p.m.
I uploaded the article to:
https://s3.amazonaws.com/ben.england/rhev-vm-rsptime.pdf
I had just gone to work for Red Hat back in 2011 so I made the mistake of including a
company-internal URL in a public-facing document. We can update the bz with the above
link if you want. This article is ANCIENT (2011), so while the basic approach should
still be relevant, some of the specifics may have changed. This was a RHEV-on-NFS
configuration.
As for why dirty ratio lowering is not part of a tuned profile, there is a debate in the
performance team about dirty ratio, with some advocating for higher dirty ratios that help
some application workloads (example: writing /tmp files). I'd stand by this article
for the particular configuration that was being studied though, and I think the basic
approach of controlling latency and improving fairness by managing queue depths is very
relevant today.
cc'ed Sanjay Rao, who continues to be a RHEV expert in the perf team (I no longer am
working on it).
-ben
----- Original Message -----
From: "Martin Polednik" <mpolednik(a)redhat.com>
To: "Sven Kieske" <s.kieske(a)mittwald.de>
Cc: devel(a)ovirt.org, "Ben" <bengland(a)redhat.com>
Sent: Wednesday, November 30, 2016 7:22:05 AM
Subject: Re: VDSM changes Linux memory dirty ratios - why?
On 30/11/16 13:10 +0100, Sven Kieske wrote:
>On 30/11/16 08:48, Martin Polednik wrote:
>> It's not really irrelevant, the host still uses disk cache. Anyway,
>> there is BZ[1] with a presentation[2] that (imho reasonably) states:
>>
>> "Reduce dirty page limits in KVM host to allow
>> direct I/O writer VMs to compete successfully
>> with buffered writer processes for storage
>> access"
>>
>> I wonder why virtual-host tuned profile doesn't contain these values:
>>
>> $ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
>> vm.dirty_background_ratio = 5
>>
>> [1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
>> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
>
>Could you share [2] with the wider community? This would be awesome!
Sorry, I've totally missed the fact it's internal link (unfortunately
publicly visible on the BZ). I believe it's slightly outdated, but
let's ask the author.
Ben, is the document[2] somehow still valid and could it be made publicly
available?
Re-referencing for completeness:
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
>--
>Mit freundlichen Grüßen / Regards
>
>Sven Kieske
>
>Systemadministrator
>Mittwald CM Service GmbH & Co. KG
>Königsberger Straße 6
>32339 Espelkamp
>T: +495772 293100
>F: +495772 293333
>https://www.mittwald.de
>Geschäftsführer: Robert Meyer
>St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
>Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
>
>_______________________________________________
>Devel mailing list
>Devel(a)ovirt.org
>http://lists.ovirt.org/mailman/listinfo/devel
12:18 p.m.
On Nov 30, 2016 4:48 PM, "Ben England" <bengland(a)redhat.com> wrote:
I uploaded the article to:
https://s3.amazonaws.com/ben.england/rhev-vm-rsptime.pdf
I had just gone to work for Red Hat back in 2011 so I made the mistake of
including a company-internal URL in a public-facing document. We can
update the bz with the above link if you want. This article is ANCIENT
(2011), so while the basic approach should still be relevant, some of the
specifics may have changed. This was a RHEV-on-NFS configuration.
As for why dirty ratio lowering is not part of a tuned profile, there is a
debate in the performance team about dirty ratio, with some advocating for
higher dirty ratios that help some application workloads (example: writing
/tmp files). I'd stand by this article for the particular configuration
that was being studied though, and I think the basic approach of
controlling latency and improving fairness by managing queue depths is very
relevant today.
cc'ed Sanjay Rao, who continues to be a RHEV expert in the perf team (I no
longer am working on it).
Ben, any idea why, at the time, you have added a workload from the host,
that was competing with the VMs?
Usually the only IO load from the hosts are the logs... Nothing serious
really.
Y.
-ben
----- Original Message -----
From: "Martin Polednik" <mpolednik(a)redhat.com>
To: "Sven Kieske" <s.kieske(a)mittwald.de>
Cc: devel(a)ovirt.org, "Ben" <bengland(a)redhat.com>
Sent: Wednesday, November 30, 2016 7:22:05 AM
Subject: Re: VDSM changes Linux memory dirty ratios - why?
On 30/11/16 13:10 +0100, Sven Kieske wrote:
>On 30/11/16 08:48, Martin Polednik wrote:
>> It's not really irrelevant, the host still uses disk cache. Anyway,
>> there is BZ[1] with a presentation[2] that (imho reasonably) states:
>>
>> "Reduce dirty page limits in KVM host to allow
>> direct I/O writer VMs to compete successfully
>> with buffered writer processes for storage
>> access"
>>
>> I wonder why virtual-host tuned profile doesn't contain these values:
>>
>> $ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
>> vm.dirty_background_ratio = 5
>>
>> [1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
>> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/
rhev/rhev-vm-rsptime.pdf
>
>Could you share [2] with the wider community? This would be awesome!
Sorry, I've totally missed the fact it's internal link (unfortunately
publicly visible on the BZ). I believe it's slightly outdated, but
let's ask the author.
Ben, is the document[2] somehow still valid and could it be made publicly
available?
Re-referencing for completeness:
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/
rhev/rhev-vm-rsptime.pdf
>--
>Mit freundlichen Grüßen / Regards
>
>Sven Kieske
>
>Systemadministrator
>Mittwald CM Service GmbH & Co. KG
>Königsberger Straße 6
>32339 Espelkamp
>T: +495772 293100
>F: +495772 293333
>https://www.mittwald.de
>Geschäftsführer: Robert Meyer
>St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad
Oeynhausen
>Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad
Oeynhausen
>
>_______________________________________________
>Devel mailing list
>Devel(a)ovirt.org
>http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2693
days inactive
2698
days old
9 comments
4 participants
participants (4)
-
Ben England -
Martin Polednik -
Sven Kieske -
Yaniv Kaul