4:57 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--WvJXSM0TULQwmuWL0sQNHhbN8U0LDFtiQ
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi,
we have some really large snapshots left from a migration. Since our
store is almost full now we have to delete them now.
Some snapshots are around 1TB already.
The last time we tried to delete a ~500GB snapshot the delete task
blocked the whole diskstore's IO and the whole cluster and all hosts
became unavailable.
Is there a way to delete large snapshots in a "humane" way that will not
block everything?
Cheers,
Stefan
--WvJXSM0TULQwmuWL0sQNHhbN8U0LDFtiQ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlROFw0ACgkQ01vvrYDXSqvzPACfUpQaEhRCNpuUwJfuwkPWV2GR
QYkAnjKzxyXlbn+ZSXWAl1SNlzZnkTvj
=KimT
-----END PGP SIGNATURE-----
--WvJXSM0TULQwmuWL0sQNHhbN8U0LDFtiQ--
5:08 a.m.
------=_NextPartTM-000-446084d0-510e-46b9-9e16-94abeca8b350
Content-Type: multipart/alternative;
boundary="_000_12EF8D94C6F8734FB2FF37B9FBEDD17358676BC5EXCHANGEcollogi_"
--_000_12EF8D94C6F8734FB2FF37B9FBEDD17358676BC5EXCHANGEcollogi_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Do you see swapping on the SPM? If yes a regular echo 3 > drop_caches could=
help.
Markus
Am 27.10.2014 10:57 schrieb Stefan Wendler <stefan.wendler(a)tngtech.com>:
Hi,
we have some really large snapshots left from a migration. Since our
store is almost full now we have to delete them now.
Some snapshots are around 1TB already.
The last time we tried to delete a ~500GB snapshot the delete task
blocked the whole diskstore's IO and the whole cluster and all hosts
became unavailable.
Is there a way to delete large snapshots in a "humane" way that will not
block everything?
Cheers,
Stefan
--_000_12EF8D94C6F8734FB2FF37B9FBEDD17358676BC5EXCHANGEcollogi_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft
Exchange Server" <!-- converted from text
--><style><!-- .EmailQuote { margin-left: 1pt; pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style </head
<body <div <p
dir=3D"ltr">Do you see swapping on the SPM? If yes a regular echo 3
>=
drop_caches could help.</p <p
dir=3D"ltr">Markus</p <div
class=3D"x_quote">Am 27.10.2014 10:57 schrieb Stefan Wendler <stefa=
n.wendler@tngtech.com>:<br type=3D"attribution" </div
</div <font
size=3D"2"><span style=3D"font-size:10pt;" <div class=3D"PlainText">Hi,<br <br we have some really large snapshots
left from a migration. Since our<br store is
almost full now we have to delete them now.<br
<br Some snapshots are around 1TB
already.<br <br The last
time we tried to delete a ~500GB snapshot the delete task<br blocked the whole diskstore's IO and the whole cluster and
all hosts<br became unavailable.<br <br Is there a way to delete large
snapshots in a "humane" way that w=
ill not<br block everything?<br <br Cheers,<br Stefan<br
<br </div </span></font
</body </html
--_000_12EF8D94C6F8734FB2FF37B9FBEDD17358676BC5EXCHANGEcollogi_--
------=_NextPartTM-000-446084d0-510e-46b9-9e16-94abeca8b350
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-446084d0-510e-46b9-9e16-94abeca8b350--
5:39 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--H13U6aH9iV7Nmtcjmrx43WbJI6iXvVa1N
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Hi,
do you mean during snapshot deletion or in general?
In general we do not have any swap usage at all. On none of the 4 hosts.
We had some swap I/O on the host that is SPM most of the time. But not
during that period where we tried to delete the snapshot.
Cheers,
Stefan
On 10/27/14 11:08, Markus Stockhausen wrote:
Do you see swapping on the SPM? If yes a regular echo 3 >
drop_caches c=
ould help.
=20
Markus
=20
Am 27.10.2014 10:57 schrieb Stefan Wendler <stefan.wendler(a)tngtech.com>=
:
Hi,
=20
we have some really large snapshots left from a migration. Since our
store is almost full now we have to delete them now.
=20
Some snapshots are around 1TB already.
=20
The last time we tried to delete a ~500GB snapshot the delete task
blocked the whole diskstore's IO and the whole cluster and all hosts
became unavailable.
=20
Is there a way to delete large snapshots in a "humane" way that will no=
t
block everything?
=20
Cheers,
Stefan
=20
=20
--H13U6aH9iV7Nmtcjmrx43WbJI6iXvVa1N
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlROIOEACgkQ01vvrYDXSquXvQCeM9oyFAcihz7G8SZNgweD4EiC
l3AAn1TGWxTLi0e4ZsNcINQKXmhZX/pz
=vDPp
-----END PGP SIGNATURE-----
--H13U6aH9iV7Nmtcjmrx43WbJI6iXvVa1N--
7:07 a.m.
This is a multi-part message in MIME format.
------=_NextPartTM-000-a0780f88-f591-414a-acb9-ba0b16afdc8d
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Von: Stefan Wendler [stefan.wendler(a)tngtech.com]=0A=
Gesendet: Montag, 27. Oktober 2014 11:39=0A=
An: Markus Stockhausen=0A=
Cc: users(a)ovirt.org=0A=
Betreff: Re: [ovirt-users] Deleting large snapshots blocks the whole clus=
ter=0A=
=0A=
Hi,=0A=
=0A=
do you mean during snapshot deletion or in general?=0A=
=0A=
In general we do not have any swap usage at all. On none of the 4 hosts.=
=0A=
=0A=
We had some swap I/O on the host that is SPM most of the time. But not=0A=
during that period where we tried to delete the snapshot.=0A=
=0A=
=0A=
I thought of the bug described in RH bugzilla 1138690 "[SCALE] snapshot del=
etion -> =0A=
heavy swapping on SPM". In case you have no access to it (as it is RHEV tag=
ged): =0A=
=0A=
qemu-img reads through pagecache during snapshot deletion. A new flag has b=
een =0A=
intrduced in August this year that allows to avoid page cache. This flag mu=
st be=0A=
provided by OVirt/RHEV. Target release is 3.6. =0A=
=0A=
Until then the only bugfix is to drop pagecache manually on regular interva=
ls on the =0A=
hypversior and especially the SPM node.=0A=
=0A=
Markus=0A=
=0A=
=0A=
Cheers,=0A=
Stefan=0A=
=0A=
On 10/27/14 11:08, Markus Stockhausen wrote:=0A=
> Do you see swapping on the SPM? If yes a regular echo 3 > drop_caches c=
ould help.=0A=
>=0A=
> Markus=0A=
>=0A=
> Am 27.10.2014 10:57 schrieb Stefan Wendler <stefan.wendler(a)tngtech.com>=
:=0A=
> Hi,=0A=
>=0A=
> we have some really large snapshots left from a migration. Since our=0A=
> store is almost full now we have to delete them now.=0A=
>=0A=
> Some snapshots are around 1TB already.=0A=
>=0A=
> The last time we tried to delete a ~500GB snapshot the delete task=0A=
blocked the whole diskstore's IO and the whole cluster and all hosts=0A=
became unavailable.=0A=
=0A=
Is there a way to delete large snapshots in a "humane" way that will not=
=0A=
block everything?=0A=
=0A=
Cheers,=0A=
Stefan=0A=
=0A=
=0A=
=0A=
------=_NextPartTM-000-a0780f88-f591-414a-acb9-ba0b16afdc8d
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-a0780f88-f591-414a-acb9-ba0b16afdc8d--
7:45 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--pRrw2Nflj1DiHOHuNiU2AGGrNfrnE9WXR
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
I see. Then I will keep an eye on this tonight where we have a huge "all
ovirt downtime" just to delete those snapshots ^^
Thanks and Cheers,
Stefan
On 10/27/14 13:07, Markus Stockhausen wrote:
> Von: Stefan Wendler [stefan.wendler(a)tngtech.com]
> Gesendet: Montag, 27. Oktober 2014 11:39
> An: Markus Stockhausen
> Cc: users(a)ovirt.org
> Betreff: Re: [ovirt-users] Deleting large snapshots blocks the whole c=
luster
>
> Hi,
>
> do you mean during snapshot deletion or in general?
>
> In general we do not have any swap usage at all. On none of the 4 host=
s.
>
> We had some swap I/O on the host that is SPM most of the time. But not=
> during that period where we tried to delete the snapshot.
>
=20
I thought of the bug described in RH bugzilla 1138690 "[SCALE] snapshot=
deletion ->=20
heavy swapping on SPM". In case you have no access to it (as it
is RHEV=
tagged):=20
=20
qemu-img reads through pagecache during snapshot deletion. A new flag h=
as
been=20
intrduced in August this year that allows to avoid page cache. This
fla=
g must be
provided by OVirt/RHEV. Target release is 3.6.=20
=20
Until then the only bugfix is to drop pagecache manually on regular int=
ervals on
the=20
hypversior and especially the SPM node.
=20
Markus
=20
>
> Cheers,
> Stefan
>
> On 10/27/14 11:08, Markus Stockhausen wrote:
>> Do you see swapping on the SPM? If yes a regular echo 3 > drop_caches=
could help.
>>
>> Markus
>>
>> Am 27.10.2014 10:57 schrieb Stefan Wendler <stefan.wendler(a)tngtech.co=
m>:
>> Hi,
>>
>> we have some really large snapshots left from a migration. Since our
>> store is almost full now we have to delete them now.
>>
>> Some snapshots are around 1TB already.
>>
>> The last time we tried to delete a ~500GB snapshot the delete task
> blocked the whole diskstore's IO and the whole cluster and all hosts
> became unavailable.
>
> Is there a way to delete large snapshots in a "humane" way that will n=
ot
> block everything?
>
> Cheers,
> Stefan
>
>
=20
--pRrw2Nflj1DiHOHuNiU2AGGrNfrnE9WXR
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlROPm0ACgkQ01vvrYDXSqsNugCgmCvjPjgrTzUIWb5OEi2QAaTJ
/NEAn1ahQxu3bb7OGpUcZgjBldYHn6st
=etB8
-----END PGP SIGNATURE-----
--pRrw2Nflj1DiHOHuNiU2AGGrNfrnE9WXR--
3709
days inactive
3709
days old
4 comments
2 participants
participants (2)
-
Markus Stockhausen -
Stefan Wendler