[Users] Snapshot merging and the effect on underlying LV metadata
--_000_7FD6C88B6AE9804AA5C2FCAEACFD1B4841943059usprmbp01c_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi I am being told that unless the "Wipe After Delete" option is set on a vDis= k, any subsequent snapshot merging of the related VM will not delete LV met= adata (or any data!) from the volume created by the snapshot. Is this correct ? I'm kinda hoping not ! Richard Davis Technical Specialist PGDS Midrange (UK) - Solaris & Linux This email is confidential and should not be used by anyone who is not the = original intended recipient. PGDS cannot accept liability for statements ma= de which are clearly the sender's own and not made on behalf of the company= . In addition no statement should be construed as giving investment advice = within or outside the United Kingdom. PGDS (UK ONE) LIMITED, Laurence Pountney Hill, London, EC4R 0HH. Incorporated and registered in England and Wales. Registered Office as above. Registered number 1967719. "PGDS" is the trading name of certain subsidiaries of Prudential plc (regis= tered in England, number 1397169), whose registered office is at Laurence P= ountney Hill London EC4R OHH, some of whose subsidiaries are authorised and= regulated, as applicable, by the Prudential Regulation Authority and the F= inancial Conduct Authority. Prudential plc is not affiliated in any manner = with Prudential Financial, Inc, a company whose principal place of business= is in the United States of America. = A list of other Prudential companies together with their registered statuto= ry details can be found in 'About Prudential' on http://www.prudential.co.uk An e-mail reply to this address may be subject to interception or monitorin= g for operational reasons or for lawful business practices. --_000_7FD6C88B6AE9804AA5C2FCAEACFD1B4841943059usprmbp01c_ Content-Type: text/html; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"> <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:"Pru Sans Normal"; panose-1:2 0 5 3 0 0 0 2 0 3;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif"; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"EN-GB" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Hi<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">I am being told that unless the “Wipe After De= lete” option is set on a vDisk, any subsequent snapshot merging of th= e related VM will not delete LV metadata (or any data!) from the volume cre= ated by the snapshot. <o:p></o:p></p> <p class=3D"MsoNormal">Is this correct ? I’m kinda hoping not != <o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><b><span style=3D"font-family:"Pru Sans Normal&= quot;;mso-fareast-language:EN-GB">Richard Davis<o:p></o:p></span></b></p> <p class=3D"MsoNormal"><span style=3D"font-size:9.0pt;font-family:"Pru= Sans Normal";mso-fareast-language:EN-GB">Technical Specialist<o:p></o= :p></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Pru Sans Normal&quo= t;;mso-fareast-language:EN-GB">PGDS Midrange (UK) – Solaris & Lin= ux<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Pru Sans Normal&quo= t;;mso-fareast-language:EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <P><br><br>This email is confidential and should not be used by anyone who = is not the original intended recipient. PGDS cannot accept liability for st= atements made which are clearly the sender's own and not made on behalf of = the company. In addition no statement should be construed as giving investm= ent advice within or outside the United Kingdom.</P> <P>PGDS (UK ONE) LIMITED, Laurence Pountney Hill, London, EC4R 0HH.<br>Inco= rporated and registered in England and Wales. Registered Office as<br>above= . Registered number 1967719.</P> <P>"PGDS" is the trading name of certain subsidiaries of Prudential plc (re= gistered in England, number 1397169), whose registered office is at Laurenc= e Pountney Hill London EC4R OHH, some of whose subsidiaries are authorised = and regulated, as applicable, by the Prudential Regulation Authority and th= e Financial Conduct Authority. Prudential plc is not affiliated in any mann= er with Prudential Financial, Inc, a company whose principal place of busin= ess is in the United States of America. <br><br>A list of other Prudential = companies together with their registered statutory details can be found in = 'About Prudential' on <A href=3D"http://www.prudential.co.uk">http://www.pr= udential.co.uk</A></P> <P>An e-mail reply to this address may be subject to interception or monito= ring for operational reasons or for lawful business practices.</P></body> </html> --_000_7FD6C88B6AE9804AA5C2FCAEACFD1B4841943059usprmbp01c_--
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 27 Feb 2014, Davis, Richard wrote:
I am being told that unless the "Wipe After Delete" option is set on a vDisk, any subsequent snapshot merging of the related VM will not delete LV metadata (or any data!) from the volume created by the snapshot. Is this correct ? I'm kinda hoping not !
It is my belief a depetion cannot be relied upon to have happened in all cases. Some options flag sets in lvm ** do ** persist old data, and so our security practice at PMman to treat data on removed LV's as though it persists There are published reports that instances on other public cloud providers have been deployed with 'non-wiped' drives in the 'slack space'. Why run the reputational risk? When we reclaim a LV, we perform a 'renaming' that permits to spot 'dirty' and 'scratched' instances needing wiping. [we also fill a new VG / PV with LV's indicating it needs wiping, as we do not wish to expose content if a drive is pulled and then re-used after testing when SMART errors appeared, but do not stand up to disqualify a drive] Later a cron driven process, sensitive to IO load runs. It builds a list of candidates over a day old, using 'find' and the LV name series showing it is dirty and scratched. Then in turn by LV found, it fires off a sub-task (when load is low), which in turn performs a 'niced' 'shred' operation on that LV, followed by the 'shred 'zeroing' operation. When load is too high, it sleeps for a couple of minutes, and re-tries fragment: $_shredCmd = "ionice -c 3 shred -n \ ".$_num_passes." -z ".$_working_lvm; Only when that sub-process has completed do we 'rename' and later 'remove' a given LV, to let its space re-enter the assignment pool - -- Russ herrold -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEARECAAYFAlMPkAMACgkQMRh1QZtklkSamQCgnVqEo2Kmzq9Ao8T0BCYhBTyn aToAoIaOVGkxX3EsVghMxOtgE3RiUr9G =rm/K -----END PGP SIGNATURE-----
A very neat solution. I like that. Thank you for sharing. -----Original Message----- From: R P Herrold [mailto:herrold@owlriver.com] Sent: 27 February 2014 19:20 To: Davis, Richard Cc: 'users@ovirt.org' Subject: Snapshot merging and the effect on underlying LV metadata -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 27 Feb 2014, Davis, Richard wrote:
I am being told that unless the "Wipe After Delete" option is set on a vDisk, any subsequent snapshot merging of the related VM will not delete LV metadata (or any data!) from the volume created by the snapshot. Is this correct ? I'm kinda hoping not !
It is my belief a depetion cannot be relied upon to have happened in all cases. Some options flag sets in lvm ** do ** persist old data, and so our security practice at PMman to treat data on removed LV's as though it persists There are published reports that instances on other public cloud providers have been deployed with 'non-wiped' drives in the 'slack space'. Why run the reputational risk? When we reclaim a LV, we perform a 'renaming' that permits to spot 'dirty' and 'scratched' instances needing wiping. [we also fill a new VG / PV with LV's indicating it needs wiping, as we do not wish to expose content if a drive is pulled and then re-used after testing when SMART errors appeared, but do not stand up to disqualify a drive] Later a cron driven process, sensitive to IO load runs. It builds a list of candidates over a day old, using 'find' and the LV name series showing it is dirty and scratched. Then in turn by LV found, it fires off a sub-task (when load is low), which in turn performs a 'niced' 'shred' operation on that LV, followed by the 'shred 'zeroing' operation. When load is too high, it sleeps for a couple of minutes, and re-tries fragment: $_shredCmd = "ionice -c 3 shred -n \ ".$_num_passes." -z ".$_working_lvm; Only when that sub-process has completed do we 'rename' and later 'remove' a given LV, to let its space re-enter the assignment pool - -- Russ herrold -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEARECAAYFAlMPkAMACgkQMRh1QZtklkSamQCgnVqEo2Kmzq9Ao8T0BCYhBTyn aToAoIaOVGkxX3EsVghMxOtgE3RiUr9G =rm/K -----END PGP SIGNATURE----- This email is confidential and should not be used by anyone who is not the original intended recipient. PGDS cannot accept liability for statements made which are clearly the sender's own and not made on behalf of the company. In addition no statement should be construed as giving investment advice within or outside the United Kingdom. PGDS (UK ONE) LIMITED, Laurence Pountney Hill, London, EC4R 0HH. Incorporated and registered in England and Wales. Registered Office as above. Registered number 1967719. "PGDS" is the trading name of certain subsidiaries of Prudential plc (registered in England, number 1397169), whose registered office is at Laurence Pountney Hill London EC4R OHH, some of whose subsidiaries are authorised and regulated, as applicable, by the Prudential Regulation Authority and the Financial Conduct Authority. Prudential plc is not affiliated in any manner with Prudential Financial, Inc, a company whose principal place of business is in the United States of America. A list of other Prudential companies together with their registered statutory details can be found in 'About Prudential' on http://www.prudential.co.uk An e-mail reply to this address may be subject to interception or monitoring for operational reasons or for lawful business practices.
In lvm2 version 2.02.105 lvconvert gained a --splitsnapshot option to allow people to wipe snapshot content before releasing the extents for reallocation. --splitsnapshot Separates SnapshotLogicalVolume from its origin. The volume that is split off contains the chunks that differ from the ori- gin along with the metadata describing them. This volume can be wiped and then destroyed with lvremove. The inverse of --snap- shot. Alasdair
On Fri, 28 Feb 2014, Alasdair G Kergon wrote:
In lvm2 version 2.02.105 lvconvert gained a --splitsnapshot option to allow people to wipe snapshot content before releasing the extents for reallocation.
--splitsnapshot Separates SnapshotLogicalVolume from its origin. The volume that is split off contains the chunks that differ from the ori- gin along with the metadata describing them. This volume can be wiped and then destroyed with lvremove. The inverse of --snap- shot.
Nice to know ... we use the snapshot feature heavilyin our virtualization, but as: CentOS 6 is at lvm2-2.02.100-8.el6.x86_64, and C 5 at lvm2-2.02.88-12.el5, we will need to wait a bit before relying on its presence. Any chance of a re-basing / refresh / backport at least into RHEL 6 (we have only one Xen oriented dom0 at this point on C5)? Thanks --Russ herrold
On Fri, Feb 28, 2014 at 02:45:28PM -0500, R P Herrold wrote:
Nice to know ... we use the snapshot feature heavilyin our virtualization, but as: CentOS 6 is at lvm2-2.02.100-8.el6.x86_64, and C 5 at lvm2-2.02.88-12.el5, we will need to wait a bit before relying on its presence. Any chance of a re-basing / refresh / backport at least into RHEL 6 (we have only one Xen oriented dom0 at this point on C5)?
It will be in RHEL6.6. Alasdair
participants (3)
-
Alasdair G Kergon -
Davis, Richard -
R P Herrold