Hi,

we just set up a new 4.1.2 OVirt cluster. It is a quite normal

HDD/XFS/NFS stack that worked quit well with 4.0 in the past.

Inside the VMs we use XFS too.

To our surprise we observe abysmal high IO during mkfs.xfs

and fstrim inside the VM. A simple example:

Step 1: Create 100G Thin disk

Result 1: Disk occupies ~10M on storage

Step 2: Format disk inside VM with mkfs.xfs

Result 2: Disk occupies 100G on storage

Changing the discard flag on the disk does not have any effect.

Am I missing something?

Best regards.

Markus

--_000_12EF8D94C6F8734FB2FF37B9FBEDD173010E01F673EXCHANGEcollo_-- ------=3D_NextPartTM-000-bc0df54e-354f-4430-b495-bb3fece85829 Content-Type: text/plain; name=3D"InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=3D"InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich erhalten haben, informieren Sie bitte sofort den Absender u= nd vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. =C3=9Cber das Internet versandte E-Mails k=C3=B6nnen unter fremden Namen er= stellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserkl=C3=A4rung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln Vorstand: Kadir Akin Dr. Michael H=C3=B6hnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht K=C3=B6ln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln executive board: Kadir Akin Dr. Michael H=C3=B6hnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=3D_NextPartTM-000-bc0df54e-354f-4430-b495-bb3fece85829-- --===============8381960838282623339== Content-Type: multipart/mixed MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tLS0tLT1fTmV4dFBhcnRUTS0wMDAtYmMwZGY1NGUtMzU0Zi00NDMwLWI0OTUtYmIzZmVjZTg1 ODI5CkNvbnRlbnQtVHlwZTogbXVsdGlwYXJ0L2FsdGVybmF0aXZlOwoJYm91bmRhcnk9Il8wMDBf MTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzMwMTBFMDFGNjczRVhDSEFOR0Vjb2xsb18i CgotLV8wMDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzMwMTBFMDFGNjczRVhDSEFO R0Vjb2xsb18KQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFyc2V0PSJpc28tODg1OS0xIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgpIaSwKCndlIGp1c3Qg c2V0IHVwIGEgbmV3IDQuMS4yIE9WaXJ0IGNsdXN0ZXIuIEl0IGlzIGEgcXVpdGUgbm9ybWFsCkhE RC9YRlMvTkZTIHN0YWNrIHRoYXQgd29ya2VkIHF1aXQgd2VsbCB3aXRoIDQuMCBpbiB0aGUgcGFz dC4KSW5zaWRlIHRoZSBWTXMgd2UgdXNlIFhGUyB0b28uCgpUbyBvdXIgc3VycHJpc2Ugd2Ugb2Jz ZXJ2ZSBhYnlzbWFsIGhpZ2ggSU8gZHVyaW5nIG1rZnMueGZzCmFuZCBmc3RyaW0gaW5zaWRlIHRo ZSBWTS4gQSBzaW1wbGUgZXhhbXBsZToKClN0ZXAgMTogQ3JlYXRlIDEwMEcgVGhpbiBkaXNrClJl c3VsdCAxOiBEaXNrIG9jY3VwaWVzIH4xME0gb24gc3RvcmFnZQoKU3RlcCAyOiBGb3JtYXQgZGlz ayBpbnNpZGUgVk0gd2l0aCBta2ZzLnhmcwpSZXN1bHQgMjogRGlzayBvY2N1cGllcyAxMDBHIG9u IHN0b3JhZ2UKCkNoYW5naW5nIHRoZSBkaXNjYXJkIGZsYWcgb24gdGhlIGRpc2sgZG9lcyBub3Qg aGF2ZSBhbnkgZWZmZWN0LgoKQW0gSSBtaXNzaW5nIHNvbWV0aGluZz8KCkJlc3QgcmVnYXJkcy4K Ck1hcmt1cwoKLS1fMDAwXzEyRUY4RDk0QzZGODczNEZCMkZGMzdCOUZCRUREMTczMDEwRTAxRjY3 M0VYQ0hBTkdFY29sbG9fCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PSJpc28tODg1 OS0xIgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgo8aHRtbCBk aXI9M0QibHRyIj4KPGhlYWQ+CjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250 ZW50PTNEInRleHQvaHRtbDsgY2hhcnNldD0zRGlzby04ODU5LT0KMSI+CjxzdHlsZSB0eXBlPTNE InRleHQvY3NzIiBpZD0zRCJvd2FQYXJhU3R5bGUiPjwvc3R5bGU+CjwvaGVhZD4KPGJvZHkgZnBz dHlsZT0zRCIxIiBvY3NpPTNEIjAiPgo8ZGl2IHN0eWxlPTNEImRpcmVjdGlvbjogbHRyO2ZvbnQt ZmFtaWx5OiBBcmlhbDtjb2xvcjogIzAwMDAwMDtmb250LXNpemU6IDE9CjBwdDsiPgo8ZGl2IHN0 eWxlPTNEImZvbnQtZmFtaWx5OlRhaG9tYTsgZm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPTNE ImZvbnQtZmFtaWx5OlRhaG9tYTsgZm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPTNEImZvbnQt ZmFtaWx5OlRhaG9tYTsgZm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPTNEImZvbnQtZmFtaWx5 OlRhaG9tYTsgZm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPTNEImZvbnQtZmFtaWx5OlRhaG9t YTsgZm9udC1zaXplOjEzcHgiPgo8ZGl2Pgo8ZGl2PkhpLCA8L2Rpdj4KPGRpdj48YnI+CjwvZGl2 Pgo8ZGl2PndlIGp1c3Qgc2V0IHVwIGEgbmV3IDQuMS4yIE9WaXJ0IGNsdXN0ZXIuIEl0IGlzIGEg cXVpdGUgbm9ybWFsPC9kaXY+CjxkaXY+SEREL1hGUy9ORlMgc3RhY2sgdGhhdCB3b3JrZWQgcXVp dCB3ZWxsIHdpdGggNC4wIGluIHRoZSBwYXN0LiZuYnNwOzwvZD0KaXY+CjxkaXY+SW5zaWRlIHRo ZSBWTXMgd2UgdXNlIFhGUyB0b28uPC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5UbyBvdXIg c3VycHJpc2Ugd2Ugb2JzZXJ2ZSBhYnlzbWFsIGhpZ2ggSU8gZHVyaW5nIG1rZnMueGZzJm5ic3A7 PC9kaXY+CjxkaXY+YW5kIGZzdHJpbSBpbnNpZGUgdGhlIFZNLiBBIHNpbXBsZSBleGFtcGxlOjwv ZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+U3RlcCAxOiBDcmVhdGUgMTAwRyBUaGluIGRpc2s8 L2Rpdj4KPGRpdj5SZXN1bHQgMTogRGlzayBvY2N1cGllcyB+MTBNIG9uIHN0b3JhZ2U8L2Rpdj4K PGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlN0ZXAgMjogRm9ybWF0IGRpc2sgaW5zaWRlIFZNIHdpdGgg bWtmcy54ZnM8L2Rpdj4KPGRpdj5SZXN1bHQgMjogRGlzayBvY2N1cGllcyAxMDBHIG9uIHN0b3Jh Z2U8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkNoYW5naW5nIHRoZSBkaXNjYXJkIGZsYWcg b24gdGhlIGRpc2sgZG9lcyBub3QgaGF2ZSBhbnkgZWZmZWN0LjwvZGl2Pgo8ZGl2Pjxicj4KPC9k aXY+CjxkaXY+QW0gSSBtaXNzaW5nIHNvbWV0aGluZz88L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8 ZGl2PkJlc3QgcmVnYXJkcy48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2Pk1hcmt1czwvZGl2 Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjxkaXYgaWQ9M0Qib2ZtZWV0LWV4 dGVuc2lvbi1pbnN0YWxsZWQiIHN0eWxlPTNEImRpc3BsYXk6bm9uZSI+PC9kaXY+CjwvZGl2Pgo8 L2Rpdj4KPC9ib2R5Pgo8L2h0bWw+CgotLV8wMDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJF REQxNzMwMTBFMDFGNjczRVhDSEFOR0Vjb2xsb18tLQoKLS0tLS0tPV9OZXh0UGFydFRNLTAwMC1i YzBkZjU0ZS0zNTRmLTQ0MzAtYjQ5NS1iYjNmZWNlODU4MjkKQ29udGVudC1UeXBlOiB0ZXh0L3Bs YWluOwoJbmFtZT0iSW50ZXJTY2FuX0Rpc2NsYWltZXIudHh0IgpDb250ZW50LVRyYW5zZmVyLUVu Y29kaW5nOiA3Yml0CkNvbnRlbnQtRGlzcG9zaXRpb246IGF0dGFjaG1lbnQ7CglmaWxlbmFtZT0i SW50ZXJTY2FuX0Rpc2NsYWltZXIudHh0IgoKKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKgpEaWVzZSBFLU1h aWwgZW50aMOkbHQgdmVydHJhdWxpY2hlIHVuZC9vZGVyIHJlY2h0bGljaCBnZXNjaMO8dHp0ZQpJ bmZvcm1hdGlvbmVuLiBXZW5uIFNpZSBuaWNodCBkZXIgcmljaHRpZ2UgQWRyZXNzYXQgc2luZCBv ZGVyIGRpZXNlIEUtTWFpbAppcnJ0w7xtbGljaCBlcmhhbHRlbiBoYWJlbiwgaW5mb3JtaWVyZW4g U2llIGJpdHRlIHNvZm9ydCBkZW4gQWJzZW5kZXIgdW5kCnZlcm5pY2h0ZW4gU2llIGRpZXNlIE1h aWwuIERhcyB1bmVybGF1YnRlIEtvcGllcmVuIHNvd2llIGRpZSB1bmJlZnVndGUKV2VpdGVyZ2Fi ZSBkaWVzZXIgTWFpbCBpc3QgbmljaHQgZ2VzdGF0dGV0LgoKw5xiZXIgZGFzIEludGVybmV0IHZl cnNhbmR0ZSBFLU1haWxzIGvDtm5uZW4gdW50ZXIgZnJlbWRlbiBOYW1lbiBlcnN0ZWxsdCBvZGVy Cm1hbmlwdWxpZXJ0IHdlcmRlbi4gRGVzaGFsYiBpc3QgZGllc2UgYWxzIEUtTWFpbCB2ZXJzY2hp Y2t0ZSBOYWNocmljaHQga2VpbmUKcmVjaHRzdmVyYmluZGxpY2hlIFdpbGxlbnNlcmtsw6RydW5n LgoKQ29sbG9naWEKVW50ZXJuZWhtZW5zYmVyYXR1bmcgQUcKVWJpZXJyaW5nIDExCkQtNTA2Nzgg S8O2bG4KClZvcnN0YW5kOgpLYWRpciBBa2luCkRyLiBNaWNoYWVsIEjDtmhuZXJiYWNoCgpWb3Jz aXR6ZW5kZXIgZGVzIEF1ZnNpY2h0c3JhdGVzOgpIYW5zIEtyaXN0aWFuIExhbmd2YQoKUmVnaXN0 ZXJnZXJpY2h0OiBBbXRzZ2VyaWNodCBLw7ZsbgpSZWdpc3Rlcm51bW1lcjogSFJCIDUyIDQ5NwoK VGhpcyBlLW1haWwgbWF5IGNvbnRhaW4gY29uZmlkZW50aWFsIGFuZC9vciBwcml2aWxlZ2VkIGlu Zm9ybWF0aW9uLiBJZiB5b3UKYXJlIG5vdCB0aGUgaW50ZW5kZWQgcmVjaXBpZW50IChvciBoYXZl IHJlY2VpdmVkIHRoaXMgZS1tYWlsIGluIGVycm9yKQpwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXIg aW1tZWRpYXRlbHkgYW5kIGRlc3Ryb3kgdGhpcyBlLW1haWwuIEFueQp1bmF1dGhvcml6ZWQgY29w eWluZywgZGlzY2xvc3VyZSBvciBkaXN0cmlidXRpb24gb2YgdGhlIG1hdGVyaWFsIGluIHRoaXMK ZS1tYWlsIGlzIHN0cmljdGx5IGZvcmJpZGRlbi4KCmUtbWFpbHMgc2VudCBvdmVyIHRoZSBpbnRl cm5ldCBtYXkgaGF2ZSBiZWVuIHdyaXR0ZW4gdW5kZXIgYSB3cm9uZyBuYW1lIG9yCmJlZW4gbWFu aXB1bGF0ZWQuIFRoYXQgaXMgd2h5IHRoaXMgbWVzc2FnZSBzZW50IGFzIGFuIGUtbWFpbCBpcyBu b3QgYQpsZWdhbGx5IGJpbmRpbmcgZGVjbGFyYXRpb24gb2YgaW50ZW50aW9uLgoKQ29sbG9naWEK VW50ZXJuZWhtZW5zYmVyYXR1bmcgQUcKVWJpZXJyaW5nIDExCkQtNTA2NzggS8O2bG4KCmV4ZWN1 dGl2ZSBib2FyZDoKS2FkaXIgQWtpbgpEci4gTWljaGFlbCBIw7ZobmVyYmFjaAoKUHJlc2lkZW50 IG9mIHRoZSBzdXBlcnZpc29yeSBib2FyZDoKSGFucyBLcmlzdGlhbiBMYW5ndmEKClJlZ2lzdHJ5 IG9mZmljZTogZGlzdHJpY3QgY291cnQgQ29sb2duZQpSZWdpc3RlciBudW1iZXI6IEhSQiA1MiA0 OTcKCioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioKCi0tLS0tLT1fTmV4dFBhcnRUTS0wMDAtYmMwZGY1NGUt MzU0Zi00NDMwLWI0OTUtYmIzZmVjZTg1ODI5LS0KCg== --===============8381960838282623339==-- From ishaby at redhat.com Sun Jun 18 06:00:48 2017 Content-Type: multipart/mixed; boundary="===============5719848173522415360==" MIME-Version: 1.0 From: Idan Shaby To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 09:00:26 +0300 Message-ID: In-Reply-To: 12EF8D94C6F8734FB2FF37B9FBEDD173010E01F673@EXCHANGE.collogia.de --===============5719848173522415360== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Markus, AFAIK, mkfs.xfs tries to discard all the blocks before formatting the device. If you don't want it to do that, you can use the "-K Do not attempt to discard blocks at mkfs time" option of mkfs.xfs. In oVirt 4.1 we introduced the "Enable Discard" flag for a virtual machine's disk. When enabled, qemu is configured to pass on live UNMAP SCSI commands from the guest to the underlying storage. If you don't need live discarding, shutdown the VM and disable the "Enable Discard" option. That will cause qemu to ignore the live UNMAP SCSI commands coming from the guest and not pass it on to the underlying storage. Note that this makes fstrim completely redundant, as the purpose of the command is to discard unused blocks under the given path. Regards, Idan On Sat, Jun 17, 2017 at 1:25 AM, Markus Stockhausen wrote: > Hi, > > we just set up a new 4.1.2 OVirt cluster. It is a quite normal > HDD/XFS/NFS stack that worked quit well with 4.0 in the past. > Inside the VMs we use XFS too. > > To our surprise we observe abysmal high IO during mkfs.xfs > and fstrim inside the VM. A simple example: > > Step 1: Create 100G Thin disk > Result 1: Disk occupies ~10M on storage > > Step 2: Format disk inside VM with mkfs.xfs > Result 2: Disk occupies 100G on storage > > Changing the discard flag on the disk does not have any effect. > > Am I missing something? > > Best regards. > > Markus > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============5719848173522415360== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2PjxkaXY+PGRpdj5IaSBNYXJrdXMsPGJyPjxicj48L2Rp dj5BRkFJSywgbWtmcy54ZnMgdHJpZXMgdG8gZGlzY2FyZCBhbGwgdGhlIGJsb2NrcyBiZWZvcmUg Zm9ybWF0dGluZyB0aGUgZGV2aWNlLjxicj5JZiB5b3UgZG9uJiMzOTt0IHdhbnQgaXQgdG8gZG8g dGhhdCwgeW91IGNhbiB1c2UgdGhlICZxdW90Oy1LwqDCoMKgwqAgRG8gbm90IGF0dGVtcHQgdG8g ZGlzY2FyZCBibG9ja3MgYXQgbWtmcyB0aW1lJnF1b3Q7IG9wdGlvbiBvZiBta2ZzLnhmcy48L2Rp dj48YnI+PC9kaXY+SW4gb1ZpcnQgNC4xIHdlIGludHJvZHVjZWQgdGhlICZxdW90O0VuYWJsZSBE aXNjYXJkJnF1b3Q7IGZsYWcgZm9yIGEgdmlydHVhbCBtYWNoaW5lJiMzOTtzIGRpc2suPGJyPjwv ZGl2PjxkaXY+V2hlbiBlbmFibGVkLCBxZW11IGlzIGNvbmZpZ3VyZWQgdG8gcGFzcyBvbiBsaXZl IFVOTUFQIFNDU0kgY29tbWFuZHMgZnJvbSB0aGUgZ3Vlc3QgdG8gdGhlIHVuZGVybHlpbmcgc3Rv cmFnZS48YnI+PC9kaXY+PGRpdj5JZiB5b3UgZG9uJiMzOTt0IG5lZWQgbGl2ZSBkaXNjYXJkaW5n LCBzaHV0ZG93biB0aGUgVk0gYW5kIGRpc2FibGUgdGhlICZxdW90O0VuYWJsZSBEaXNjYXJkJnF1 b3Q7IG9wdGlvbi4gVGhhdCB3aWxsIGNhdXNlIHFlbXUgdG8gaWdub3JlIHRoZSBsaXZlIFVOTUFQ IFNDU0kgY29tbWFuZHMgY29taW5nIGZyb20gdGhlIGd1ZXN0IGFuZCBub3QgcGFzcyBpdCBvbiB0 byB0aGUgdW5kZXJseWluZyBzdG9yYWdlLjxicj48L2Rpdj48ZGl2Pk5vdGUgdGhhdCB0aGlzIG1h a2VzIGZzdHJpbSBjb21wbGV0ZWx5IHJlZHVuZGFudCwgYXMgdGhlIHB1cnBvc2Ugb2YgdGhlIGNv bW1hbmQgaXMgdG8gZGlzY2FyZCB1bnVzZWQgYmxvY2tzIHVuZGVyIHRoZSBnaXZlbiBwYXRoLjwv ZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnIgY2xlYXI9ImFsbCI+PGRpdj48ZGl2IGNs YXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEtc21hcnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPjxk aXYgZGlyPSJsdHIiPjxkaXY+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2IGRpcj0ibHRyIj48ZGl2 Pjxicj5SZWdhcmRzLDxicj48L2Rpdj5JZGFuPGJyPjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pjwv ZGl2PjwvZGl2PjwvZGl2Pgo8YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPk9uIFNhdCwgSnVu IDE3LCAyMDE3IGF0IDE6MjUgQU0sIE1hcmt1cyBTdG9ja2hhdXNlbiA8c3BhbiBkaXI9Imx0ciI+ Jmx0OzxhIGhyZWY9Im1haWx0bzpzdG9ja2hhdXNlbkBjb2xsb2dpYS5kZSIgdGFyZ2V0PSJfYmxh bmsiPnN0b2NraGF1c2VuQGNvbGxvZ2lhLmRlPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48Ymxv Y2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3Jk ZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KCgoKCjxkaXY+CjxkaXYg c3R5bGU9ImRpcmVjdGlvbjpsdHI7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDtmb250 LXNpemU6MTBwdCI+CjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OlRhaG9tYTtmb250LXNpemU6MTNw eCI+CjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OlRhaG9tYTtmb250LXNpemU6MTNweCI+CjxkaXYg c3R5bGU9ImZvbnQtZmFtaWx5OlRhaG9tYTtmb250LXNpemU6MTNweCI+CjxkaXYgc3R5bGU9ImZv bnQtZmFtaWx5OlRhaG9tYTtmb250LXNpemU6MTNweCI+CjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5 OlRhaG9tYTtmb250LXNpemU6MTNweCI+CjxkaXY+CjxkaXY+SGksIDwvZGl2Pgo8ZGl2Pjxicj4K PC9kaXY+CjxkaXY+d2UganVzdCBzZXQgdXAgYSBuZXcgNC4xLjIgT1ZpcnQgY2x1c3Rlci4gSXQg aXMgYSBxdWl0ZSBub3JtYWw8L2Rpdj4KPGRpdj5IREQvWEZTL05GUyBzdGFjayB0aGF0IHdvcmtl ZCBxdWl0IHdlbGwgd2l0aCA0LjAgaW4gdGhlIHBhc3QuwqA8L2Rpdj4KPGRpdj5JbnNpZGUgdGhl IFZNcyB3ZSB1c2UgWEZTIHRvby48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlRvIG91ciBz dXJwcmlzZSB3ZSBvYnNlcnZlIGFieXNtYWwgaGlnaCBJTyBkdXJpbmcgbWtmcy54ZnPCoDwvZGl2 Pgo8ZGl2PmFuZCBmc3RyaW0gaW5zaWRlIHRoZSBWTS4gQSBzaW1wbGUgZXhhbXBsZTo8L2Rpdj4K PGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlN0ZXAgMTogQ3JlYXRlIDEwMEcgVGhpbiBkaXNrPC9kaXY+ CjxkaXY+UmVzdWx0IDE6IERpc2sgb2NjdXBpZXMgfjEwTSBvbiBzdG9yYWdlPC9kaXY+CjxkaXY+ PGJyPgo8L2Rpdj4KPGRpdj5TdGVwIDI6IEZvcm1hdCBkaXNrIGluc2lkZSBWTSB3aXRoIG1rZnMu eGZzPC9kaXY+CjxkaXY+UmVzdWx0IDI6IERpc2sgb2NjdXBpZXMgMTAwRyBvbiBzdG9yYWdlPC9k aXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5DaGFuZ2luZyB0aGUgZGlzY2FyZCBmbGFnIG9uIHRo ZSBkaXNrIGRvZXMgbm90IGhhdmUgYW55IGVmZmVjdC48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8 ZGl2PkFtIEkgbWlzc2luZyBzb21ldGhpbmc/PC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5C ZXN0IHJlZ2FyZHMuPC9kaXY+PHNwYW4gY2xhc3M9IkhPRW5aYiI+PGZvbnQgY29sb3I9IiM4ODg4 ODgiPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+TWFya3VzPC9kaXY+CjwvZm9udD48L3NwYW4+PC9k aXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8ZGl2IGlkPSJtXzM5NzUwNTY1NzA4NTgx NTE4NjZvZm1lZXQtZXh0ZW5zaW9uLWluc3RhbGxlZCIgc3R5bGU9ImRpc3BsYXk6bm9uZSI+PC9k aXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+Cgo8YnI+X19fX19fX19fX19fX19fX19fX19fX19fX19f X19fPHdicj5fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBo cmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBo cmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHJlbD0i bm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvPHdicj5t YWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KPGJyPjwvYmxvY2txdW90ZT48L2Rpdj48YnI+ PC9kaXY+PC9kaXY+Cg== --===============5719848173522415360==-- From nsoffer at redhat.com Sun Jun 18 06:29:23 2017 Content-Type: multipart/mixed; boundary="===============8607691802564825024==" MIME-Version: 1.0 From: Nir Soffer To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 06:29:11 +0000 Message-ID: In-Reply-To: CAO8Of3m01daR7tTavuZYgNC0qkDxUMNkMN4THfQ=ypsK_DY8Rw@mail.gmail.com --===============8607691802564825024== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=90=D7=B3, 18 = =D7=91=D7=99=D7=95=D7=A0=D7=99 2017, 9:01, =D7=9E=D7=90=D7=AA Idan Shaby = =E2=80=8F: > Hi Markus, > > AFAIK, mkfs.xfs tries to discard all the blocks before formatting the > device. > If you don't want it to do that, you can use the "-K Do not attempt to > discard blocks at mkfs time" option of mkfs.xfs. > > In oVirt 4.1 we introduced the "Enable Discard" flag for a virtual > machine's disk. > When enabled, qemu is configured to pass on live UNMAP SCSI commands from > the guest to the underlying storage. > If you don't need live discarding, shutdown the VM and disable the "Enable > Discard" option. That will cause qemu to ignore the live UNMAP SCSI > commands coming from the guest and not pass it on to the underlying stora= ge. > Note that this makes fstrim completely redundant, as the purpose of the > command is to discard unused blocks under the given path. > I think we need a bug for this, both fo documrnting this issue, and to investigate why discarding unused blocks allocate and zero all blocks. This behaviour is unhelpful. Markus, can you check if performing the same discard from the host leads to same result? > > Regards, > Idan > > On Sat, Jun 17, 2017 at 1:25 AM, Markus Stockhausen < > stockhausen(a)collogia.de> wrote: > >> Hi, >> >> we just set up a new 4.1.2 OVirt cluster. It is a quite normal >> HDD/XFS/NFS stack that worked quit well with 4.0 in the past. >> Inside the VMs we use XFS too. >> >> To our surprise we observe abysmal high IO during mkfs.xfs >> and fstrim inside the VM. A simple example: >> >> Step 1: Create 100G Thin disk >> Result 1: Disk occupies ~10M on storage >> >> Step 2: Format disk inside VM with mkfs.xfs >> Result 2: Disk occupies 100G on storage >> >> Changing the discard flag on the disk does not have any effect. >> >> Am I missing something? >> >> Best regards. >> >> Markus >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > --===============8607691802564825024== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGJyPjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBkaXI9Imx0ciI+15HXqteQ16jX mdeaINeZ15XXnSDXkNezLCAxOCDXkdeZ15XXoNeZIDIwMTcsIDk6MDEsINee15DXqiBJZGFuIFNo YWJ5IOKAjyZsdDs8YSBocmVmPSJtYWlsdG86aXNoYWJ5QHJlZGhhdC5jb20iPmlzaGFieUByZWRo YXQuY29tPC9hPiZndDs6PGJyPjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIg c3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRp bmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPjxkaXY+PGRpdj48ZGl2PjxkaXY+SGkgTWFya3Vz LDxicj48YnI+PC9kaXY+QUZBSUssIG1rZnMueGZzIHRyaWVzIHRvIGRpc2NhcmQgYWxsIHRoZSBi bG9ja3MgYmVmb3JlIGZvcm1hdHRpbmcgdGhlIGRldmljZS48YnI+SWYgeW91IGRvbiYjMzk7dCB3 YW50IGl0IHRvIGRvIHRoYXQsIHlvdSBjYW4gdXNlIHRoZSAmcXVvdDstS8KgwqDCoMKgIERvIG5v dCBhdHRlbXB0IHRvIGRpc2NhcmQgYmxvY2tzIGF0IG1rZnMgdGltZSZxdW90OyBvcHRpb24gb2Yg bWtmcy54ZnMuPC9kaXY+PGJyPjwvZGl2PkluIG9WaXJ0IDQuMSB3ZSBpbnRyb2R1Y2VkIHRoZSAm cXVvdDtFbmFibGUgRGlzY2FyZCZxdW90OyBmbGFnIGZvciBhIHZpcnR1YWwgbWFjaGluZSYjMzk7 cyBkaXNrLjxicj48L2Rpdj48ZGl2PldoZW4gZW5hYmxlZCwgcWVtdSBpcyBjb25maWd1cmVkIHRv IHBhc3Mgb24gbGl2ZSBVTk1BUCBTQ1NJIGNvbW1hbmRzIGZyb20gdGhlIGd1ZXN0IHRvIHRoZSB1 bmRlcmx5aW5nIHN0b3JhZ2UuPGJyPjwvZGl2PjxkaXY+SWYgeW91IGRvbiYjMzk7dCBuZWVkIGxp dmUgZGlzY2FyZGluZywgc2h1dGRvd24gdGhlIFZNIGFuZCBkaXNhYmxlIHRoZSAmcXVvdDtFbmFi bGUgRGlzY2FyZCZxdW90OyBvcHRpb24uIFRoYXQgd2lsbCBjYXVzZSBxZW11IHRvIGlnbm9yZSB0 aGUgbGl2ZSBVTk1BUCBTQ1NJIGNvbW1hbmRzIGNvbWluZyBmcm9tIHRoZSBndWVzdCBhbmQgbm90 IHBhc3MgaXQgb24gdG8gdGhlIHVuZGVybHlpbmcgc3RvcmFnZS48YnI+PC9kaXY+PGRpdj5Ob3Rl IHRoYXQgdGhpcyBtYWtlcyBmc3RyaW0gY29tcGxldGVseSByZWR1bmRhbnQsIGFzIHRoZSBwdXJw b3NlIG9mIHRoZSBjb21tYW5kIGlzIHRvIGRpc2NhcmQgdW51c2VkIGJsb2NrcyB1bmRlciB0aGUg Z2l2ZW4gcGF0aC48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj5JIHRoaW5rIHdlIG5lZWQgYSBidWcgZm9yIHRoaXMsIGJvdGggZm8gZG9jdW1ybnRpbmcg dGhpcyBpc3N1ZSwgYW5kIHRvIGludmVzdGlnYXRlIHdoeSBkaXNjYXJkaW5nIHVudXNlZCBibG9j a3MgYWxsb2NhdGUgYW5kIHplcm8gYWxsIGJsb2Nrcy4gVGhpcyBiZWhhdmlvdXIgaXMgdW5oZWxw ZnVsLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+TWFya3VzLCBjYW4geW91IGNoZWNrIGlmIHBl cmZvcm1pbmcgdGhlIHNhbWUgZGlzY2FyZCBmcm9tIHRoZSBob3N0IGxlYWRzIHRvIHNhbWUgcmVz dWx0PzwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXYgY2xhc3M9ImdtYWls X3F1b3RlIj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAw IDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2 IGRpcj0ibHRyIj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyIGNsZWFyPSJhbGwiPjxkaXY+ PGRpdiBjbGFzcz0ibV83ODcxOTc3NDg2NzcxMzQzNDYyZ21haWxfc2lnbmF0dXJlIiBkYXRhLXNt YXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj48ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXYgZGlyPSJs dHIiPjxkaXY+PGRpdiBkaXI9Imx0ciI+PGRpdj48YnI+UmVnYXJkcyw8YnI+PC9kaXY+SWRhbjxi cj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48 ZGl2IGRpcj0ibHRyIj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+Cjxicj48ZGl2IGNsYXNzPSJn bWFpbF9xdW90ZSI+T24gU2F0LCBKdW4gMTcsIDIwMTcgYXQgMToyNSBBTSwgTWFya3VzIFN0b2Nr aGF1c2VuIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOnN0b2NraGF1c2VuQGNv bGxvZ2lhLmRlIiB0YXJnZXQ9Il9ibGFuayI+c3RvY2toYXVzZW5AY29sbG9naWEuZGU8L2E+Jmd0 Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9 Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVm dDoxZXgiPgoKCgoKPGRpdj4KPGRpdiBzdHlsZT0iZGlyZWN0aW9uOmx0cjtmb250LWZhbWlseTpB cmlhbDtjb2xvcjojMDAwMDAwO2ZvbnQtc2l6ZToxMHB0Ij4KPGRpdiBzdHlsZT0iZm9udC1mYW1p bHk6VGFob21hO2ZvbnQtc2l6ZToxM3B4Ij4KPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6VGFob21h O2ZvbnQtc2l6ZToxM3B4Ij4KPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6VGFob21hO2ZvbnQtc2l6 ZToxM3B4Ij4KPGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6VGFob21hO2ZvbnQtc2l6ZToxM3B4Ij4K PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6VGFob21hO2ZvbnQtc2l6ZToxM3B4Ij4KPGRpdj4KPGRp dj5IaSwgPC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj53ZSBqdXN0IHNldCB1cCBhIG5ldyA0 LjEuMiBPVmlydCBjbHVzdGVyLiBJdCBpcyBhIHF1aXRlIG5vcm1hbDwvZGl2Pgo8ZGl2PkhERC9Y RlMvTkZTIHN0YWNrIHRoYXQgd29ya2VkIHF1aXQgd2VsbCB3aXRoIDQuMCBpbiB0aGUgcGFzdC7C oDwvZGl2Pgo8ZGl2Pkluc2lkZSB0aGUgVk1zIHdlIHVzZSBYRlMgdG9vLjwvZGl2Pgo8ZGl2Pjxi cj4KPC9kaXY+CjxkaXY+VG8gb3VyIHN1cnByaXNlIHdlIG9ic2VydmUgYWJ5c21hbCBoaWdoIElP IGR1cmluZyBta2ZzLnhmc8KgPC9kaXY+CjxkaXY+YW5kIGZzdHJpbSBpbnNpZGUgdGhlIFZNLiBB IHNpbXBsZSBleGFtcGxlOjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+U3RlcCAxOiBDcmVh dGUgMTAwRyBUaGluIGRpc2s8L2Rpdj4KPGRpdj5SZXN1bHQgMTogRGlzayBvY2N1cGllcyB+MTBN IG9uIHN0b3JhZ2U8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlN0ZXAgMjogRm9ybWF0IGRp c2sgaW5zaWRlIFZNIHdpdGggbWtmcy54ZnM8L2Rpdj4KPGRpdj5SZXN1bHQgMjogRGlzayBvY2N1 cGllcyAxMDBHIG9uIHN0b3JhZ2U8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkNoYW5naW5n IHRoZSBkaXNjYXJkIGZsYWcgb24gdGhlIGRpc2sgZG9lcyBub3QgaGF2ZSBhbnkgZWZmZWN0Ljwv ZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+QW0gSSBtaXNzaW5nIHNvbWV0aGluZz88L2Rpdj4K PGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkJlc3QgcmVnYXJkcy48L2Rpdj48c3BhbiBjbGFzcz0ibV83 ODcxOTc3NDg2NzcxMzQzNDYySE9FblpiIj48Zm9udCBjb2xvcj0iIzg4ODg4OCI+CjxkaXY+PGJy Pgo8L2Rpdj4KPGRpdj5NYXJrdXM8L2Rpdj4KPC9mb250Pjwvc3Bhbj48L2Rpdj4KPC9kaXY+Cjwv ZGl2Pgo8L2Rpdj4KPC9kaXY+CjxkaXYgaWQ9Im1fNzg3MTk3NzQ4Njc3MTM0MzQ2Mm1fMzk3NTA1 NjU3MDg1ODE1MTg2Nm9mbWVldC1leHRlbnNpb24taW5zdGFsbGVkIiBzdHlsZT0iZGlzcGxheTpu b25lIj48L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KCjxicj5fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8 YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3Zp cnQub3JnPC9hPjxicj4KPGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xp c3RpbmZvL3VzZXJzIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlz dHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+PC9ibG9ja3F1 b3RlPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFp bHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48 YnI+CjxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vy cyIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KPC9ibG9ja3F1b3RlPjwvZGl2Pgo= --===============8607691802564825024==-- From ykaul at redhat.com Sun Jun 18 07:59:30 2017 Content-Type: multipart/mixed; boundary="===============8214399089084795831==" MIME-Version: 1.0 From: Yaniv Kaul To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 10:58:47 +0300 Message-ID: In-Reply-To: 12EF8D94C6F8734FB2FF37B9FBEDD173010E01F673@EXCHANGE.collogia.de --===============8214399089084795831== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Sat, Jun 17, 2017 at 1:25 AM, Markus Stockhausen wrote: > Hi, > > we just set up a new 4.1.2 OVirt cluster. It is a quite normal > HDD/XFS/NFS stack that worked quit well with 4.0 in the past. > Inside the VMs we use XFS too. > > To our surprise we observe abysmal high IO during mkfs.xfs > and fstrim inside the VM. A simple example: > > Step 1: Create 100G Thin disk > Result 1: Disk occupies ~10M on storage > > Step 2: Format disk inside VM with mkfs.xfs > Result 2: Disk occupies 100G on storage > > Changing the discard flag on the disk does not have any effect. > Are you sure it's discarding, at all? 1. NFS: only NFSv4.2 supports discard. Is that the case in your setup? 2. What's the value of /sys/block//queue/discard_granularity ? 3. Can you share the mkfs.xfs command line? 4. Are you sure it's not a raw-sparse image? Y. > Am I missing something? > > Best regards. > > Markus > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============8214399089084795831== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIFNhdCwgSnVuIDE3LCAyMDE3IGF0IDE6MjUgQU0sIE1hcmt1cyBT dG9ja2hhdXNlbiA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpzdG9ja2hhdXNl bkBjb2xsb2dpYS5kZSIgdGFyZ2V0PSJfYmxhbmsiPnN0b2NraGF1c2VuQGNvbGxvZ2lhLmRlPC9h PiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0 eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigy MDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+CgoKCgo8ZGl2Pgo8ZGl2IHN0eWxlPSJkaXJl Y3Rpb246bHRyO2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOnJnYigwLDAsMCk7Zm9udC1zaXplOjEw cHQiPgo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTpUYWhvbWE7Zm9udC1zaXplOjEzcHgiPgo8ZGl2 IHN0eWxlPSJmb250LWZhbWlseTpUYWhvbWE7Zm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPSJm b250LWZhbWlseTpUYWhvbWE7Zm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPSJmb250LWZhbWls eTpUYWhvbWE7Zm9udC1zaXplOjEzcHgiPgo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTpUYWhvbWE7 Zm9udC1zaXplOjEzcHgiPgo8ZGl2Pgo8ZGl2PkhpLCA8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8 ZGl2PndlIGp1c3Qgc2V0IHVwIGEgbmV3IDQuMS4yIE9WaXJ0IGNsdXN0ZXIuIEl0IGlzIGEgcXVp dGUgbm9ybWFsPC9kaXY+CjxkaXY+SEREL1hGUy9ORlMgc3RhY2sgdGhhdCB3b3JrZWQgcXVpdCB3 ZWxsIHdpdGggNC4wIGluIHRoZSBwYXN0LsKgPC9kaXY+CjxkaXY+SW5zaWRlIHRoZSBWTXMgd2Ug dXNlIFhGUyB0b28uPC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5UbyBvdXIgc3VycHJpc2Ug d2Ugb2JzZXJ2ZSBhYnlzbWFsIGhpZ2ggSU8gZHVyaW5nIG1rZnMueGZzwqA8L2Rpdj4KPGRpdj5h bmQgZnN0cmltIGluc2lkZSB0aGUgVk0uIEEgc2ltcGxlIGV4YW1wbGU6PC9kaXY+CjxkaXY+PGJy Pgo8L2Rpdj4KPGRpdj5TdGVwIDE6IENyZWF0ZSAxMDBHIFRoaW4gZGlzazwvZGl2Pgo8ZGl2PlJl c3VsdCAxOiBEaXNrIG9jY3VwaWVzIH4xME0gb24gc3RvcmFnZTwvZGl2Pgo8ZGl2Pjxicj4KPC9k aXY+CjxkaXY+U3RlcCAyOiBGb3JtYXQgZGlzayBpbnNpZGUgVk0gd2l0aCBta2ZzLnhmczwvZGl2 Pgo8ZGl2PlJlc3VsdCAyOiBEaXNrIG9jY3VwaWVzIDEwMEcgb24gc3RvcmFnZTwvZGl2Pgo8ZGl2 Pjxicj4KPC9kaXY+CjxkaXY+Q2hhbmdpbmcgdGhlIGRpc2NhcmQgZmxhZyBvbiB0aGUgZGlzayBk b2VzIG5vdCBoYXZlIGFueSBlZmZlY3QuPC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9k aXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+QXJl IHlvdSBzdXJlIGl0JiMzOTtzIGRpc2NhcmRpbmcsIGF0IGFsbD88L2Rpdj48ZGl2PjEuIE5GUzog b25seSBORlN2NC4yIHN1cHBvcnRzIGRpc2NhcmQuIElzIHRoYXQgdGhlIGNhc2UgaW4geW91ciBz ZXR1cD88L2Rpdj48ZGl2PjIuIFdoYXQmIzM5O3MgdGhlIHZhbHVlIG9mwqAvc3lzL2Jsb2NrLyZs dDtkaXNrJmd0Oy9xdWV1ZS9kaXNjYXJkX2dyYW51bGFyaXR5ID88L2Rpdj48ZGl2PjMuIENhbiB5 b3Ugc2hhcmUgdGhlIG1rZnMueGZzIGNvbW1hbmQgbGluZT88L2Rpdj48ZGl2PjQuIEFyZSB5b3Ug c3VyZSBpdCYjMzk7cyBub3QgYSByYXctc3BhcnNlIGltYWdlPzwvZGl2PjxkaXY+WS48L2Rpdj48 ZGl2Pjxicj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJn aW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIw NCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdj48ZGl2IHN0eWxlPSJkaXJlY3Rpb246bHRyO2ZvbnQt ZmFtaWx5OkFyaWFsO2NvbG9yOnJnYigwLDAsMCk7Zm9udC1zaXplOjEwcHQiPjxkaXYgc3R5bGU9 ImZvbnQtZmFtaWx5OlRhaG9tYTtmb250LXNpemU6MTNweCI+PGRpdiBzdHlsZT0iZm9udC1mYW1p bHk6VGFob21hO2ZvbnQtc2l6ZToxM3B4Ij48ZGl2IHN0eWxlPSJmb250LWZhbWlseTpUYWhvbWE7 Zm9udC1zaXplOjEzcHgiPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OlRhaG9tYTtmb250LXNpemU6 MTNweCI+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6VGFob21hO2ZvbnQtc2l6ZToxM3B4Ij48ZGl2 Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+QW0gSSBtaXNzaW5nIHNvbWV0aGluZz88L2Rpdj4KPGRp dj48YnI+CjwvZGl2Pgo8ZGl2PkJlc3QgcmVnYXJkcy48L2Rpdj48c3BhbiBjbGFzcz0iZ21haWwt SE9FblpiIj48Zm9udCBjb2xvcj0iIzg4ODg4OCI+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5NYXJr dXM8L2Rpdj4KPC9mb250Pjwvc3Bhbj48L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+ CjxkaXYgaWQ9ImdtYWlsLW1fLTgxNDYxNTQwMzMxMjkyMDkyMTFvZm1lZXQtZXh0ZW5zaW9uLWlu c3RhbGxlZCIgc3R5bGU9ImRpc3BsYXk6bm9uZSI+PC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+ Cgo8YnI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj5fX19fX19fX19fX19fX19f Xzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQu b3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQu b3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxh bmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9h Pjxicj4KPGJyPjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+PC9kaXY+Cg== --===============8214399089084795831==-- From fabrice.bacchella at orange.fr Sun Jun 18 08:14:03 2017 Content-Type: multipart/mixed; boundary="===============3904565740771521015==" MIME-Version: 1.0 From: Fabrice Bacchella To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 10:13:57 +0200 Message-ID: In-Reply-To: CAO8Of3m01daR7tTavuZYgNC0qkDxUMNkMN4THfQ=ypsK_DY8Rw@mail.gmail.com --===============3904565740771521015== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable > Le 18 juin 2017 =C3=A0 08:00, Idan Shaby a =C3=A9cr= it : > If you don't need live discarding, shutdown the VM and disable the "Enabl= e Discard" option. That will cause qemu to ignore the live UNMAP SCSI comma= nds coming from the guest and not pass it on to the underlying storage. > Note that this makes fstrim completely redundant, as the purpose of the c= ommand is to discard unused blocks under the given path. Redundant ? Useless you mean ? From my comprehension, the purpose to fstrim= is to send UNMAP SCSI on batch instead of mount -o discard that send them = synchronously. --===============3904565740771521015==-- From stockhausen at collogia.de Sun Jun 18 08:16:34 2017 Content-Type: multipart/mixed; boundary="===============7495127685692045517==" MIME-Version: 1.0 From: Markus Stockhausen To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 08:16:31 +0000 Message-ID: <12EF8D94C6F8734FB2FF37B9FBEDD173010E01F845@EXCHANGE.collogia.de> In-Reply-To: B61C53CC-C476-4ADA-8DF6-55A3B9976369@orange.fr --===============7495127685692045517== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. ------=3D_NextPartTM-000-bb13544b-1435-41ab-bb19-431fe0d6715d Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Thanks for all your feedback.=3D0A=3D =3D0A=3D Im trying to collect all the infos in BZ1462504.=3D0A=3D =3D0A=3D ________________________________________=3D0A=3D Von: Fabrice Bacchella [fabrice.bacchella(a)orange.fr]=3D0A=3D Gesendet: Sonntag, 18. Juni 2017 10:13=3D0A=3D An: Idan Shaby=3D0A=3D Cc: Markus Stockhausen; Ovirt Users=3D0A=3D Betreff: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contra= =3D productive=3D0A=3D =3D0A=3D > Le 18 juin 2017 =3DE0 08:00, Idan Shaby a =3DE9crit= :=3D0A=3D > If you don't need live discarding, shutdown the VM and disable the "Enabl= =3D e Discard" option. That will cause qemu to ignore the live UNMAP SCSI comma= =3D nds coming from the guest and not pass it on to the underlying storage.=3D0= A=3D > Note that this makes fstrim completely redundant, as the purpose of the c= =3D ommand is to discard unused blocks under the given path.=3D0A=3D =3D0A=3D Redundant ? Useless you mean ? From my comprehension, the purpose to fstrim= =3D is to send UNMAP SCSI on batch instead of mount -o discard that send them = =3D synchronously.=3D0A=3D ------=3D_NextPartTM-000-bb13544b-1435-41ab-bb19-431fe0d6715d Content-Type: text/plain; name=3D"InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=3D"InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich erhalten haben, informieren Sie bitte sofort den Absender u= nd vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. =C3=9Cber das Internet versandte E-Mails k=C3=B6nnen unter fremden Namen er= stellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserkl=C3=A4rung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln Vorstand: Kadir Akin Dr. Michael H=C3=B6hnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht K=C3=B6ln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln executive board: Kadir Akin Dr. Michael H=C3=B6hnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=3D_NextPartTM-000-bb13544b-1435-41ab-bb19-431fe0d6715d-- --===============7495127685692045517== Content-Type: multipart/mixed MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" ClRoaXMgaXMgYSBtdWx0aS1wYXJ0IG1lc3NhZ2UgaW4gTUlNRSBmb3JtYXQuCgotLS0tLS09X05l eHRQYXJ0VE0tMDAwLWJiMTM1NDRiLTE0MzUtNDFhYi1iYjE5LTQzMWZlMGQ2NzE1ZApDb250ZW50 LVR5cGU6IHRleHQvcGxhaW47IGNoYXJzZXQ9Imlzby04ODU5LTEiCkNvbnRlbnQtVHJhbnNmZXIt RW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKClRoYW5rcyBmb3IgYWxsIHlvdXIgZmVlZGJhY2su PTBBPQo9MEE9CkltIHRyeWluZyB0byBjb2xsZWN0IGFsbCB0aGUgaW5mb3MgaW4gQloxNDYyNTA0 Lj0wQT0KPTBBPQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPTBBPQpW b246IEZhYnJpY2UgQmFjY2hlbGxhIFtmYWJyaWNlLmJhY2NoZWxsYUBvcmFuZ2UuZnJdPTBBPQpH ZXNlbmRldDogU29ubnRhZywgMTguIEp1bmkgMjAxNyAxMDoxMz0wQT0KQW46IElkYW4gU2hhYnk9 MEE9CkNjOiBNYXJrdXMgU3RvY2toYXVzZW47IE92aXJ0IFVzZXJzPTBBPQpCZXRyZWZmOiBSZTog W292aXJ0LXVzZXJzXSBPVmlydCA0LjEuMiAtIHRyaW0vZGlzY2FyZCBvbiBIREQvWEZTL05GUyBj b250cmE9CnByb2R1Y3RpdmU9MEE9Cj0wQT0KPiBMZSAxOCBqdWluIDIwMTcgPUUwIDA4OjAwLCBJ ZGFuIFNoYWJ5IDxpc2hhYnlAcmVkaGF0LmNvbT4gYSA9RTljcml0IDo9MEE9Cj4gSWYgeW91IGRv bid0IG5lZWQgbGl2ZSBkaXNjYXJkaW5nLCBzaHV0ZG93biB0aGUgVk0gYW5kIGRpc2FibGUgdGhl ICJFbmFibD0KZSBEaXNjYXJkIiBvcHRpb24uIFRoYXQgd2lsbCBjYXVzZSBxZW11IHRvIGlnbm9y ZSB0aGUgbGl2ZSBVTk1BUCBTQ1NJIGNvbW1hPQpuZHMgY29taW5nIGZyb20gdGhlIGd1ZXN0IGFu ZCBub3QgcGFzcyBpdCBvbiB0byB0aGUgdW5kZXJseWluZyBzdG9yYWdlLj0wQT0KPiBOb3RlIHRo YXQgdGhpcyBtYWtlcyBmc3RyaW0gY29tcGxldGVseSByZWR1bmRhbnQsIGFzIHRoZSBwdXJwb3Nl IG9mIHRoZSBjPQpvbW1hbmQgaXMgdG8gZGlzY2FyZCB1bnVzZWQgYmxvY2tzIHVuZGVyIHRoZSBn aXZlbiBwYXRoLj0wQT0KPTBBPQpSZWR1bmRhbnQgPyBVc2VsZXNzIHlvdSBtZWFuID8gRnJvbSBt eSBjb21wcmVoZW5zaW9uLCB0aGUgcHVycG9zZSB0byBmc3RyaW09CiBpcyB0byBzZW5kIFVOTUFQ IFNDU0kgb24gYmF0Y2ggaW5zdGVhZCBvZiBtb3VudCAtbyBkaXNjYXJkIHRoYXQgc2VuZCB0aGVt ID0Kc3luY2hyb25vdXNseS49MEE9Ci0tLS0tLT1fTmV4dFBhcnRUTS0wMDAtYmIxMzU0NGItMTQz NS00MWFiLWJiMTktNDMxZmUwZDY3MTVkCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsKCW5hbWU9 IkludGVyU2Nhbl9EaXNjbGFpbWVyLnR4dCIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogN2Jp dApDb250ZW50LURpc3Bvc2l0aW9uOiBhdHRhY2htZW50OwoJZmlsZW5hbWU9IkludGVyU2Nhbl9E aXNjbGFpbWVyLnR4dCIKCioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioKRGllc2UgRS1NYWlsIGVudGjDpGx0 IHZlcnRyYXVsaWNoZSB1bmQvb2RlciByZWNodGxpY2ggZ2VzY2jDvHR6dGUKSW5mb3JtYXRpb25l bi4gV2VubiBTaWUgbmljaHQgZGVyIHJpY2h0aWdlIEFkcmVzc2F0IHNpbmQgb2RlciBkaWVzZSBF LU1haWwKaXJydMO8bWxpY2ggZXJoYWx0ZW4gaGFiZW4sIGluZm9ybWllcmVuIFNpZSBiaXR0ZSBz b2ZvcnQgZGVuIEFic2VuZGVyIHVuZAp2ZXJuaWNodGVuIFNpZSBkaWVzZSBNYWlsLiBEYXMgdW5l cmxhdWJ0ZSBLb3BpZXJlbiBzb3dpZSBkaWUgdW5iZWZ1Z3RlCldlaXRlcmdhYmUgZGllc2VyIE1h aWwgaXN0IG5pY2h0IGdlc3RhdHRldC4KCsOcYmVyIGRhcyBJbnRlcm5ldCB2ZXJzYW5kdGUgRS1N YWlscyBrw7ZubmVuIHVudGVyIGZyZW1kZW4gTmFtZW4gZXJzdGVsbHQgb2RlcgptYW5pcHVsaWVy dCB3ZXJkZW4uIERlc2hhbGIgaXN0IGRpZXNlIGFscyBFLU1haWwgdmVyc2NoaWNrdGUgTmFjaHJp Y2h0IGtlaW5lCnJlY2h0c3ZlcmJpbmRsaWNoZSBXaWxsZW5zZXJrbMOkcnVuZy4KCkNvbGxvZ2lh ClVudGVybmVobWVuc2JlcmF0dW5nIEFHClViaWVycmluZyAxMQpELTUwNjc4IEvDtmxuCgpWb3Jz dGFuZDoKS2FkaXIgQWtpbgpEci4gTWljaGFlbCBIw7ZobmVyYmFjaAoKVm9yc2l0emVuZGVyIGRl cyBBdWZzaWNodHNyYXRlczoKSGFucyBLcmlzdGlhbiBMYW5ndmEKClJlZ2lzdGVyZ2VyaWNodDog QW10c2dlcmljaHQgS8O2bG4KUmVnaXN0ZXJudW1tZXI6IEhSQiA1MiA0OTcKClRoaXMgZS1tYWls IG1heSBjb250YWluIGNvbmZpZGVudGlhbCBhbmQvb3IgcHJpdmlsZWdlZCBpbmZvcm1hdGlvbi4g SWYgeW91CmFyZSBub3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCAob3IgaGF2ZSByZWNlaXZlZCB0 aGlzIGUtbWFpbCBpbiBlcnJvcikKcGxlYXNlIG5vdGlmeSB0aGUgc2VuZGVyIGltbWVkaWF0ZWx5 IGFuZCBkZXN0cm95IHRoaXMgZS1tYWlsLiBBbnkKdW5hdXRob3JpemVkIGNvcHlpbmcsIGRpc2Ns b3N1cmUgb3IgZGlzdHJpYnV0aW9uIG9mIHRoZSBtYXRlcmlhbCBpbiB0aGlzCmUtbWFpbCBpcyBz dHJpY3RseSBmb3JiaWRkZW4uCgplLW1haWxzIHNlbnQgb3ZlciB0aGUgaW50ZXJuZXQgbWF5IGhh dmUgYmVlbiB3cml0dGVuIHVuZGVyIGEgd3JvbmcgbmFtZSBvcgpiZWVuIG1hbmlwdWxhdGVkLiBU aGF0IGlzIHdoeSB0aGlzIG1lc3NhZ2Ugc2VudCBhcyBhbiBlLW1haWwgaXMgbm90IGEKbGVnYWxs eSBiaW5kaW5nIGRlY2xhcmF0aW9uIG9mIGludGVudGlvbi4KCkNvbGxvZ2lhClVudGVybmVobWVu c2JlcmF0dW5nIEFHClViaWVycmluZyAxMQpELTUwNjc4IEvDtmxuCgpleGVjdXRpdmUgYm9hcmQ6 CkthZGlyIEFraW4KRHIuIE1pY2hhZWwgSMO2aG5lcmJhY2gKClByZXNpZGVudCBvZiB0aGUgc3Vw ZXJ2aXNvcnkgYm9hcmQ6CkhhbnMgS3Jpc3RpYW4gTGFuZ3ZhCgpSZWdpc3RyeSBvZmZpY2U6IGRp c3RyaWN0IGNvdXJ0IENvbG9nbmUKUmVnaXN0ZXIgbnVtYmVyOiBIUkIgNTIgNDk3CgoqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqCgotLS0tLS09X05leHRQYXJ0VE0tMDAwLWJiMTM1NDRiLTE0MzUtNDFhYi1i YjE5LTQzMWZlMGQ2NzE1ZC0tCg== --===============7495127685692045517==-- From ishaby at redhat.com Sun Jun 18 08:19:00 2017 Content-Type: multipart/mixed; boundary="===============7833101812622719347==" MIME-Version: 1.0 From: Idan Shaby To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 11:18:38 +0300 Message-ID: In-Reply-To: B61C53CC-C476-4ADA-8DF6-55A3B9976369@orange.fr --===============7833101812622719347== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Right, but I just wanted to emphasize that disabling "Enable Discard" for that disk will cause qemu to ignore these UNMAP commands and not pass it on to the underlying storage. So if you've got this flag disabled, there's no reason to use fstrim. It makes sense to use it only when enabling "Discard enabled". Regards, Idan On Sun, Jun 18, 2017 at 11:13 AM, Fabrice Bacchella < fabrice.bacchella(a)orange.fr> wrote: > > > Le 18 juin 2017 =C3=A0 08:00, Idan Shaby a =C3=A9= crit : > > If you don't need live discarding, shutdown the VM and disable the > "Enable Discard" option. That will cause qemu to ignore the live UNMAP SC= SI > commands coming from the guest and not pass it on to the underlying stora= ge. > > Note that this makes fstrim completely redundant, as the purpose of the > command is to discard unused blocks under the given path. > > Redundant ? Useless you mean ? From my comprehension, the purpose to > fstrim is to send UNMAP SCSI on batch instead of mount -o discard that se= nd > them synchronously. --===============7833101812622719347== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2PlJpZ2h0LCBidXQgSSBqdXN0IHdhbnRlZCB0byBlbXBo YXNpemUgdGhhdCBkaXNhYmxpbmcgJnF1b3Q7RW5hYmxlIERpc2NhcmQmcXVvdDsgZm9yIHRoYXQg ZGlzayB3aWxsIGNhdXNlIHFlbXUgdG8gaWdub3JlIHRoZXNlIFVOTUFQIGNvbW1hbmRzIGFuZCBu b3QgcGFzcyBpdCBvbiB0byB0aGUgdW5kZXJseWluZyBzdG9yYWdlLjxicj48L2Rpdj5TbyBpZiB5 b3UmIzM5O3ZlIGdvdCB0aGlzIGZsYWcgZGlzYWJsZWQsIHRoZXJlJiMzOTtzIG5vIHJlYXNvbiB0 byB1c2UgZnN0cmltLiBJdCBtYWtlcyBzZW5zZSB0byB1c2UgaXQgb25seSB3aGVuIGVuYWJsaW5n ICZxdW90O0Rpc2NhcmQgZW5hYmxlZCZxdW90Oy48YnI+PC9kaXY+PC9kaXY+PGRpdiBjbGFzcz0i Z21haWxfZXh0cmEiPjxiciBjbGVhcj0iYWxsIj48ZGl2PjxkaXYgY2xhc3M9ImdtYWlsX3NpZ25h dHVyZSIgZGF0YS1zbWFydG1haWw9ImdtYWlsX3NpZ25hdHVyZSI+PGRpdiBkaXI9Imx0ciI+PGRp dj48ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXYgZGlyPSJsdHIiPjxkaXY+PGJyPlJlZ2FyZHMsPGJy PjwvZGl2PklkYW48YnI+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+ Cjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gU3VuLCBKdW4gMTgsIDIwMTcgYXQgMTE6 MTMgQU0sIEZhYnJpY2UgQmFjY2hlbGxhIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFp bHRvOmZhYnJpY2UuYmFjY2hlbGxhQG9yYW5nZS5mciIgdGFyZ2V0PSJfYmxhbmsiPmZhYnJpY2Uu YmFjY2hlbGxhQG9yYW5nZS5mcjwvYT4mZ3Q7PC9zcGFuPiB3cm90ZTo8YnI+PGJsb2NrcXVvdGUg Y2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6 MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PHNwYW4gY2xhc3M9IiI+PGJyPgomZ3Q7 IExlIDE4IGp1aW4gMjAxNyDDoCAwODowMCwgSWRhbiBTaGFieSAmbHQ7PGEgaHJlZj0ibWFpbHRv OmlzaGFieUByZWRoYXQuY29tIj5pc2hhYnlAcmVkaGF0LmNvbTwvYT4mZ3Q7IGEgw6ljcml0IDo8 YnI+CiZndDsgSWYgeW91IGRvbiYjMzk7dCBuZWVkIGxpdmUgZGlzY2FyZGluZywgc2h1dGRvd24g dGhlIFZNIGFuZCBkaXNhYmxlIHRoZSAmcXVvdDtFbmFibGUgRGlzY2FyZCZxdW90OyBvcHRpb24u IFRoYXQgd2lsbCBjYXVzZSBxZW11IHRvIGlnbm9yZSB0aGUgbGl2ZSBVTk1BUCBTQ1NJIGNvbW1h bmRzIGNvbWluZyBmcm9tIHRoZSBndWVzdCBhbmQgbm90IHBhc3MgaXQgb24gdG8gdGhlIHVuZGVy bHlpbmcgc3RvcmFnZS48YnI+CiZndDsgTm90ZSB0aGF0IHRoaXMgbWFrZXMgZnN0cmltIGNvbXBs ZXRlbHkgcmVkdW5kYW50LCBhcyB0aGUgcHVycG9zZSBvZiB0aGUgY29tbWFuZCBpcyB0byBkaXNj YXJkIHVudXNlZCBibG9ja3MgdW5kZXIgdGhlIGdpdmVuIHBhdGguPGJyPgo8YnI+Cjwvc3Bhbj5S ZWR1bmRhbnQgPyBVc2VsZXNzIHlvdSBtZWFuID8gRnJvbSBteSBjb21wcmVoZW5zaW9uLCB0aGUg cHVycG9zZSB0byBmc3RyaW0gaXMgdG8gc2VuZCBVTk1BUCBTQ1NJIG9uIGJhdGNoIGluc3RlYWQg b2YgbW91bnQgLW8gZGlzY2FyZCB0aGF0IHNlbmQgdGhlbSBzeW5jaHJvbm91c2x5LjwvYmxvY2tx dW90ZT48L2Rpdj48YnI+PC9kaXY+Cg== --===============7833101812622719347==-- From stockhausen at collogia.de Sun Jun 18 08:31:07 2017 Content-Type: multipart/mixed; boundary="===============1961673493606483215==" MIME-Version: 1.0 From: Markus Stockhausen To: users at ovirt.org Subject: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS contraproductive Date: Sun, 18 Jun 2017 08:31:04 +0000 Message-ID: <12EF8D94C6F8734FB2FF37B9FBEDD173010E01F85C@EXCHANGE.collogia.de> In-Reply-To: CAJgorsY==y9a_9cCLZDuTnnOudf8mx+mzgwvPp5SsqNVc5n7JQ@mail.gmail.com --===============1961673493606483215== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ------=3D_NextPartTM-000-9c273f1c-a2dd-4aba-ab90-4aa24b5ef225 Content-Type: multipart/alternative; boundary=3D"_000_12EF8D94C6F8734FB2FF37B9FBEDD173010E01F85CEXCHANGEcollo_" --_000_12EF8D94C6F8734FB2FF37B9FBEDD173010E01F85CEXCHANGEcollo_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable > Von: Yaniv Kaul [ykaul(a)redhat.com] > Gesendet: Sonntag, 18. Juni 2017 09:58 > An: Markus Stockhausen > Cc: Ovirt Users > Betreff: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XFS/NFS cont= =3D raproductive On Sat, Jun 17, 2017 at 1:25 AM, Markus Stockhausen > wrote: Hi, we just set up a new 4.1.2 OVirt cluster. It is a quite normal HDD/XFS/NFS stack that worked quit well with 4.0 in the past. Inside the VMs we use XFS too. To our surprise we observe abysmal high IO during mkfs.xfs and fstrim inside the VM. A simple example: Step 1: Create 100G Thin disk Result 1: Disk occupies ~10M on storage Step 2: Format disk inside VM with mkfs.xfs Result 2: Disk occupies 100G on storage Changing the discard flag on the disk does not have any effect. > Are you sure it's discarding, at all? > 1. NFS: only NFSv4.2 supports discard. Is that the case in your setup? > 2. What's the value of /sys/block//queue/discard_granularity ? > 3. Can you share the mkfs.xfs command line? > 4. Are you sure it's not a raw-sparse image? Questions should answered in BZ1462504. When talking about thin provisioned disks I'm only referring to the OVirt disk-option. So I might mix up something here. Nevertheless the following is more than strange for me: - Create disk image: File on storage is small - Format inside VM: File on storage is fully allocated - Move around in Ovirt to another NFS storage: File is small again. That means: - mkfs.xfs inside VM and so qemu is hammering (empty) data into all blocks - But this data must be zeros as they can be compated afterwards. Best regards. Markus --_000_12EF8D94C6F8734FB2FF37B9FBEDD173010E01F85CEXCHANGEcollo_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable

> Von: Yaniv Kaul [ykaul(a)redhat.= com] > Gesendet: Sonntag, 18. Juni 2017 09:58
> An: Markus Stockhausen
> Cc: Ovirt Users
> Betreff: Re: [ovirt-users] OVirt 4.1.2 - trim/discard on HDD/XF= =3D S/NFS contraproductive

On Sat, Jun 17, 2017 at 1:25 AM, Markus Stockh= au=3D sen <stoc= khause=3D n(a)collogia.de> wrote:

Hi,

we just set up a new 4.1.2 OVirt cluster. It is a quite normal

HDD/XFS/NFS stack that worked quit well with 4.0 in the past.
Inside the VMs we use XFS too.

To our surprise we observe abysmal high IO during mkfs.xfs

and fstrim inside the VM. A simple example:

Step 1: Create 100G Thin disk

Result 1: Disk occupies ~10M on storage

Step 2: Format disk inside VM with mkfs.xfs

Result 2: Disk occupies 100G on storage

Changing the discard flag on the disk does not have any effect.

> Are you sure it's discarding, at all?

> 1. NFS: only NFSv4.2 supports discard. Is that the case in your s= =3D etup?

> 2. What's the value of /sys/block/<disk>/queue/discard= =3D _granularity ?

> 3. Can you share the mkfs.xfs command line?

> 4. Are you sure it's not a raw-sparse image?

Questions should answered in BZ1462504. When talking about thin

provisioned disks I'm only referring to the OVirt disk-option. So I

might mix up something here. Nevertheless the following is more than

strange for me:

- Create disk image: File on storage is small

- Format inside VM: File on storage is fully allocated

- Move around in Ovirt to another NFS storage: File is small again.

That means:

- mkfs.xfs inside VM and so qemu is hammering (empty) data into all bl= =3D ocks

- But this data must be zeros as they can be compated afterwards.

Best regards.

Markus

--_000_12EF8D94C6F8734FB2FF37B9FBEDD173010E01F85CEXCHANGEcollo_-- ------=3D_NextPartTM-000-9c273f1c-a2dd-4aba-ab90-4aa24b5ef225 Content-Type: text/plain; name=3D"InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=3D"InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich erhalten haben, informieren Sie bitte sofort den Absender u= nd vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. =C3=9Cber das Internet versandte E-Mails k=C3=B6nnen unter fremden Namen er= stellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserkl=C3=A4rung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln Vorstand: Kadir Akin Dr. Michael H=C3=B6hnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht K=C3=B6ln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 K=C3=B6ln executive board: Kadir Akin Dr. Michael H=C3=B6hnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=3D_NextPartTM-000-9c273f1c-a2dd-4aba-ab90-4aa24b5ef225-- --===============1961673493606483215== Content-Type: multipart/mixed MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tLS0tLT1fTmV4dFBhcnRUTS0wMDAtOWMyNzNmMWMtYTJkZC00YWJhLWFiOTAtNGFhMjRiNWVm MjI1CkNvbnRlbnQtVHlwZTogbXVsdGlwYXJ0L2FsdGVybmF0aXZlOwoJYm91bmRhcnk9Il8wMDBf MTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzMwMTBFMDFGODVDRVhDSEFOR0Vjb2xsb18i CgotLV8wMDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzMwMTBFMDFGODVDRVhDSEFO R0Vjb2xsb18KQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFyc2V0PSJpc28tODg1OS0xIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgo+IFZvbjogWWFuaXYg S2F1bCBbeWthdWxAcmVkaGF0LmNvbV0KPiBHZXNlbmRldDogU29ubnRhZywgMTguIEp1bmkgMjAx NyAwOTo1OAo+IEFuOiBNYXJrdXMgU3RvY2toYXVzZW4KPiBDYzogT3ZpcnQgVXNlcnMKPiBCZXRy ZWZmOiBSZTogW292aXJ0LXVzZXJzXSBPVmlydCA0LjEuMiAtIHRyaW0vZGlzY2FyZCBvbiBIREQv WEZTL05GUyBjb250PQpyYXByb2R1Y3RpdmUKCk9uIFNhdCwgSnVuIDE3LCAyMDE3IGF0IDE6MjUg QU0sIE1hcmt1cyBTdG9ja2hhdXNlbiA8c3RvY2toYXVzZW5AY29sbG9naWEuZD0KZTxtYWlsdG86 c3RvY2toYXVzZW5AY29sbG9naWEuZGU+PiB3cm90ZToKSGksCgp3ZSBqdXN0IHNldCB1cCBhIG5l dyA0LjEuMiBPVmlydCBjbHVzdGVyLiBJdCBpcyBhIHF1aXRlIG5vcm1hbApIREQvWEZTL05GUyBz dGFjayB0aGF0IHdvcmtlZCBxdWl0IHdlbGwgd2l0aCA0LjAgaW4gdGhlIHBhc3QuCkluc2lkZSB0 aGUgVk1zIHdlIHVzZSBYRlMgdG9vLgoKVG8gb3VyIHN1cnByaXNlIHdlIG9ic2VydmUgYWJ5c21h bCBoaWdoIElPIGR1cmluZyBta2ZzLnhmcwphbmQgZnN0cmltIGluc2lkZSB0aGUgVk0uIEEgc2lt cGxlIGV4YW1wbGU6CgpTdGVwIDE6IENyZWF0ZSAxMDBHIFRoaW4gZGlzawpSZXN1bHQgMTogRGlz ayBvY2N1cGllcyB+MTBNIG9uIHN0b3JhZ2UKClN0ZXAgMjogRm9ybWF0IGRpc2sgaW5zaWRlIFZN IHdpdGggbWtmcy54ZnMKUmVzdWx0IDI6IERpc2sgb2NjdXBpZXMgMTAwRyBvbiBzdG9yYWdlCgpD aGFuZ2luZyB0aGUgZGlzY2FyZCBmbGFnIG9uIHRoZSBkaXNrIGRvZXMgbm90IGhhdmUgYW55IGVm ZmVjdC4KCj4gQXJlIHlvdSBzdXJlIGl0J3MgZGlzY2FyZGluZywgYXQgYWxsPwo+IDEuIE5GUzog b25seSBORlN2NC4yIHN1cHBvcnRzIGRpc2NhcmQuIElzIHRoYXQgdGhlIGNhc2UgaW4geW91ciBz ZXR1cD8KPiAyLiBXaGF0J3MgdGhlIHZhbHVlIG9mIC9zeXMvYmxvY2svPGRpc2s+L3F1ZXVlL2Rp c2NhcmRfZ3JhbnVsYXJpdHkgPwo+IDMuIENhbiB5b3Ugc2hhcmUgdGhlIG1rZnMueGZzIGNvbW1h bmQgbGluZT8KPiA0LiBBcmUgeW91IHN1cmUgaXQncyBub3QgYSByYXctc3BhcnNlIGltYWdlPwoK UXVlc3Rpb25zIHNob3VsZCBhbnN3ZXJlZCBpbiBCWjE0NjI1MDQuIFdoZW4gdGFsa2luZyBhYm91 dCB0aGluCnByb3Zpc2lvbmVkIGRpc2tzIEknbSBvbmx5IHJlZmVycmluZyB0byB0aGUgT1ZpcnQg ZGlzay1vcHRpb24uIFNvIEkKbWlnaHQgbWl4IHVwIHNvbWV0aGluZyBoZXJlLiBOZXZlcnRoZWxl c3MgdGhlIGZvbGxvd2luZyBpcyBtb3JlIHRoYW4Kc3RyYW5nZSBmb3IgbWU6CgotIENyZWF0ZSBk aXNrIGltYWdlOiBGaWxlIG9uIHN0b3JhZ2UgaXMgc21hbGwKLSBGb3JtYXQgaW5zaWRlIFZNOiBG aWxlIG9uIHN0b3JhZ2UgaXMgZnVsbHkgYWxsb2NhdGVkCi0gTW92ZSBhcm91bmQgaW4gT3ZpcnQg dG8gYW5vdGhlciBORlMgc3RvcmFnZTogRmlsZSBpcyBzbWFsbCBhZ2Fpbi4KClRoYXQgbWVhbnM6 Ci0gbWtmcy54ZnMgaW5zaWRlIFZNIGFuZCBzbyBxZW11IGlzIGhhbW1lcmluZyAoZW1wdHkpIGRh dGEgaW50byBhbGwgYmxvY2tzCi0gQnV0IHRoaXMgZGF0YSBtdXN0IGJlIHplcm9zIGFzIHRoZXkg Y2FuIGJlIGNvbXBhdGVkIGFmdGVyd2FyZHMuCgpCZXN0IHJlZ2FyZHMuCgpNYXJrdXMKCgotLV8w MDBfMTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzMwMTBFMDFGODVDRVhDSEFOR0Vjb2xs b18KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9Imlzby04ODU5LTEiCkNvbnRlbnQt VHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCjxodG1sIGRpcj0zRCJsdHIiPgo8 aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0zRCJDb250ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9o dG1sOyBjaGFyc2V0PTNEaXNvLTg4NTktPQoxIj4KPHN0eWxlIHR5cGU9M0QidGV4dC9jc3MiIGlk PTNEIm93YVBhcmFTdHlsZSI+PC9zdHlsZT4KPC9oZWFkPgo8Ym9keSBmcHN0eWxlPTNEIjEiIG9j c2k9M0QiMCI+CjxkaXYgc3R5bGU9M0QiZGlyZWN0aW9uOiBsdHI7Zm9udC1mYW1pbHk6IEFyaWFs O2NvbG9yOiAjMDAwMDAwO2ZvbnQtc2l6ZTogMT0KMHB0OyI+CjxkaXYgc3R5bGU9M0QiZm9udC1m YW1pbHk6IFRpbWVzIE5ldyBSb21hbjsgY29sb3I6ICMwMDAwMDA7IGZvbnQtc2l6ZTogMTZweD0K Ij4KPGRpdiBpZD0zRCJkaXZScEYzMzExMzciIHN0eWxlPTNEImRpcmVjdGlvbjogbHRyOyI+PGZv bnQgZmFjZT0zRCJUYWhvbWEiIHNpPQp6ZT0zRCIyIiBjb2xvcj0zRCIjMDAwMDAwIj48Yj4mZ3Q7 IFZvbjo8L2I+IFlhbml2IEthdWwgW3lrYXVsQHJlZGhhdC5jb21dPGI9CnI+CjxiPiZndDsgR2Vz ZW5kZXQ6PC9iPiBTb25udGFnLCAxOC4gSnVuaSAyMDE3IDA5OjU4PGJyPgo8Yj4mZ3Q7IEFuOjwv Yj4gTWFya3VzIFN0b2NraGF1c2VuPGJyPgo8Yj4mZ3Q7IENjOjwvYj4gT3ZpcnQgVXNlcnM8YnI+ CjxiPiZndDsgQmV0cmVmZjo8L2I+IFJlOiBbb3ZpcnQtdXNlcnNdIE9WaXJ0IDQuMS4yIC0gdHJp bS9kaXNjYXJkIG9uIEhERC9YRj0KUy9ORlMgY29udHJhcHJvZHVjdGl2ZTxicj4KPC9mb250Pjwv ZGl2Pgo8ZGl2IGlkPTNEImRpdlJwRjMzMTEzNyIgc3R5bGU9M0QiZGlyZWN0aW9uOiBsdHI7Ij48 Zm9udCBmYWNlPTNEIlRhaG9tYSIgc2k9CnplPTNEIjIiIGNvbG9yPTNEIiMwMDAwMDAiPjxicj4K PC9mb250PjwvZGl2Pgo8ZGl2Pgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2IGNsYXNzPTNEImdtYWls X2V4dHJhIj4KPGRpdiBjbGFzcz0zRCJnbWFpbF9xdW90ZSI+T24gU2F0LCBKdW4gMTcsIDIwMTcg YXQgMToyNSBBTSwgTWFya3VzIFN0b2NraGF1PQpzZW4gPHNwYW4gZGlyPTNEImx0ciI+CiZsdDs8 YSBocmVmPTNEIm1haWx0bzpzdG9ja2hhdXNlbkBjb2xsb2dpYS5kZSIgdGFyZ2V0PTNEIl9ibGFu ayI+c3RvY2toYXVzZT0KbkBjb2xsb2dpYS5kZTwvYT4mZ3Q7PC9zcGFuPiB3cm90ZTo8YnI+Cjxi bG9ja3F1b3RlIGNsYXNzPTNEImdtYWlsX3F1b3RlIiBzdHlsZT0zRCJtYXJnaW46MHB4IDBweCAw cHggMC44ZXg7IGJvcmRlcj0KLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7IHBhZGRp bmctbGVmdDoxZXgiPgo8ZGl2Pgo8ZGl2IHN0eWxlPTNEImRpcmVjdGlvbjpsdHI7IGZvbnQtZmFt aWx5OkFyaWFsOyBjb2xvcjpyZ2IoMCwwLDApOyBmb250LXNpemU9CjoxMHB0Ij4KPGRpdiBzdHls ZT0zRCJmb250LWZhbWlseTpUYWhvbWE7IGZvbnQtc2l6ZToxM3B4Ij4KPGRpdiBzdHlsZT0zRCJm b250LWZhbWlseTpUYWhvbWE7IGZvbnQtc2l6ZToxM3B4Ij4KPGRpdiBzdHlsZT0zRCJmb250LWZh bWlseTpUYWhvbWE7IGZvbnQtc2l6ZToxM3B4Ij4KPGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTpU YWhvbWE7IGZvbnQtc2l6ZToxM3B4Ij4KPGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTpUYWhvbWE7 IGZvbnQtc2l6ZToxM3B4Ij4KPGRpdj4KPGRpdj5IaSwgPC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4K PGRpdj53ZSBqdXN0IHNldCB1cCBhIG5ldyA0LjEuMiBPVmlydCBjbHVzdGVyLiBJdCBpcyBhIHF1 aXRlIG5vcm1hbDwvZGl2Pgo8ZGl2PkhERC9YRlMvTkZTIHN0YWNrIHRoYXQgd29ya2VkIHF1aXQg d2VsbCB3aXRoIDQuMCBpbiB0aGUgcGFzdC4mbmJzcDs8L2Q9Cml2Pgo8ZGl2Pkluc2lkZSB0aGUg Vk1zIHdlIHVzZSBYRlMgdG9vLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+VG8gb3VyIHN1 cnByaXNlIHdlIG9ic2VydmUgYWJ5c21hbCBoaWdoIElPIGR1cmluZyBta2ZzLnhmcyZuYnNwOzwv ZGl2Pgo8ZGl2PmFuZCBmc3RyaW0gaW5zaWRlIHRoZSBWTS4gQSBzaW1wbGUgZXhhbXBsZTo8L2Rp dj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlN0ZXAgMTogQ3JlYXRlIDEwMEcgVGhpbiBkaXNrPC9k aXY+CjxkaXY+UmVzdWx0IDE6IERpc2sgb2NjdXBpZXMgfjEwTSBvbiBzdG9yYWdlPC9kaXY+Cjxk aXY+PGJyPgo8L2Rpdj4KPGRpdj5TdGVwIDI6IEZvcm1hdCBkaXNrIGluc2lkZSBWTSB3aXRoIG1r ZnMueGZzPC9kaXY+CjxkaXY+UmVzdWx0IDI6IERpc2sgb2NjdXBpZXMgMTAwRyBvbiBzdG9yYWdl PC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5DaGFuZ2luZyB0aGUgZGlzY2FyZCBmbGFnIG9u IHRoZSBkaXNrIGRvZXMgbm90IGhhdmUgYW55IGVmZmVjdC48L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8 L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Jsb2NrcXVvdGU+Cjxk aXY+PGJyPgo8L2Rpdj4KPGRpdj4mZ3Q7IEFyZSB5b3Ugc3VyZSBpdCdzIGRpc2NhcmRpbmcsIGF0 IGFsbD88L2Rpdj4KPGRpdj4mZ3Q7IDEuIE5GUzogb25seSBORlN2NC4yIHN1cHBvcnRzIGRpc2Nh cmQuIElzIHRoYXQgdGhlIGNhc2UgaW4geW91ciBzPQpldHVwPzwvZGl2Pgo8ZGl2PiZndDsgMi4g V2hhdCdzIHRoZSB2YWx1ZSBvZiZuYnNwOy9zeXMvYmxvY2svJmx0O2Rpc2smZ3Q7L3F1ZXVlL2Rp c2NhcmQ9Cl9ncmFudWxhcml0eSA/PC9kaXY+CjxkaXY+Jmd0OyAzLiBDYW4geW91IHNoYXJlIHRo ZSBta2ZzLnhmcyBjb21tYW5kIGxpbmU/PC9kaXY+CjxkaXY+Jmd0OyA0LiBBcmUgeW91IHN1cmUg aXQncyBub3QgYSByYXctc3BhcnNlIGltYWdlPzwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+ UXVlc3Rpb25zIHNob3VsZCBhbnN3ZXJlZCBpbiBCWjE0NjI1MDQuIFdoZW4gdGFsa2luZyBhYm91 dCB0aGluPC9kaXY+CjxkaXY+cHJvdmlzaW9uZWQgZGlza3MgSSdtIG9ubHkgcmVmZXJyaW5nIHRv IHRoZSBPVmlydCBkaXNrLW9wdGlvbi4gU28gSTwvZD0KaXY+CjxkaXY+bWlnaHQgbWl4IHVwIHNv bWV0aGluZyBoZXJlLiBOZXZlcnRoZWxlc3MgdGhlIGZvbGxvd2luZyBpcyBtb3JlIHRoYW48Lz0K ZGl2Pgo8ZGl2PnN0cmFuZ2UgZm9yIG1lOjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+LSBD cmVhdGUgZGlzayBpbWFnZTogRmlsZSBvbiBzdG9yYWdlIGlzIHNtYWxsPC9kaXY+CjxkaXY+LSBG b3JtYXQgaW5zaWRlIFZNOiBGaWxlIG9uIHN0b3JhZ2UgaXMgZnVsbHkgYWxsb2NhdGVkJm5ic3A7 PC9kaXY+CjxkaXY+LSBNb3ZlIGFyb3VuZCBpbiBPdmlydCB0byBhbm90aGVyIE5GUyBzdG9yYWdl OiBGaWxlIGlzIHNtYWxsIGFnYWluLiA8Lz0KZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+VGhh dCBtZWFuczo8L2Rpdj4KPGRpdj4tIG1rZnMueGZzIGluc2lkZSBWTSBhbmQgc28gcWVtdSBpcyBo YW1tZXJpbmcgKGVtcHR5KSBkYXRhIGludG8gYWxsIGJsPQpvY2tzPC9kaXY+CjxkaXY+LSBCdXQg dGhpcyBkYXRhIG11c3QgYmUgemVyb3MgYXMgdGhleSBjYW4gYmUgY29tcGF0ZWQgYWZ0ZXJ3YXJk cy48L2Rpdj0KPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+QmVzdCByZWdhcmRzLjwvZGl2Pgo8ZGl2 Pjxicj4KPC9kaXY+CjxkaXY+TWFya3VzPC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPC9kaXY+Cjwv ZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9ib2R5Pgo8L2h0bWw+CgotLV8wMDBf MTJFRjhEOTRDNkY4NzM0RkIyRkYzN0I5RkJFREQxNzMwMTBFMDFGODVDRVhDSEFOR0Vjb2xsb18t LQoKLS0tLS0tPV9OZXh0UGFydFRNLTAwMC05YzI3M2YxYy1hMmRkLTRhYmEtYWI5MC00YWEyNGI1 ZWYyMjUKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOwoJbmFtZT0iSW50ZXJTY2FuX0Rpc2NsYWlt ZXIudHh0IgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA3Yml0CkNvbnRlbnQtRGlzcG9zaXRp b246IGF0dGFjaG1lbnQ7CglmaWxlbmFtZT0iSW50ZXJTY2FuX0Rpc2NsYWltZXIudHh0IgoKKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKgpEaWVzZSBFLU1haWwgZW50aMOkbHQgdmVydHJhdWxpY2hlIHVuZC9v ZGVyIHJlY2h0bGljaCBnZXNjaMO8dHp0ZQpJbmZvcm1hdGlvbmVuLiBXZW5uIFNpZSBuaWNodCBk ZXIgcmljaHRpZ2UgQWRyZXNzYXQgc2luZCBvZGVyIGRpZXNlIEUtTWFpbAppcnJ0w7xtbGljaCBl cmhhbHRlbiBoYWJlbiwgaW5mb3JtaWVyZW4gU2llIGJpdHRlIHNvZm9ydCBkZW4gQWJzZW5kZXIg dW5kCnZlcm5pY2h0ZW4gU2llIGRpZXNlIE1haWwuIERhcyB1bmVybGF1YnRlIEtvcGllcmVuIHNv d2llIGRpZSB1bmJlZnVndGUKV2VpdGVyZ2FiZSBkaWVzZXIgTWFpbCBpc3QgbmljaHQgZ2VzdGF0 dGV0LgoKw5xiZXIgZGFzIEludGVybmV0IHZlcnNhbmR0ZSBFLU1haWxzIGvDtm5uZW4gdW50ZXIg ZnJlbWRlbiBOYW1lbiBlcnN0ZWxsdCBvZGVyCm1hbmlwdWxpZXJ0IHdlcmRlbi4gRGVzaGFsYiBp c3QgZGllc2UgYWxzIEUtTWFpbCB2ZXJzY2hpY2t0ZSBOYWNocmljaHQga2VpbmUKcmVjaHRzdmVy YmluZGxpY2hlIFdpbGxlbnNlcmtsw6RydW5nLgoKQ29sbG9naWEKVW50ZXJuZWhtZW5zYmVyYXR1 bmcgQUcKVWJpZXJyaW5nIDExCkQtNTA2NzggS8O2bG4KClZvcnN0YW5kOgpLYWRpciBBa2luCkRy LiBNaWNoYWVsIEjDtmhuZXJiYWNoCgpWb3JzaXR6ZW5kZXIgZGVzIEF1ZnNpY2h0c3JhdGVzOgpI YW5zIEtyaXN0aWFuIExhbmd2YQoKUmVnaXN0ZXJnZXJpY2h0OiBBbXRzZ2VyaWNodCBLw7ZsbgpS ZWdpc3Rlcm51bW1lcjogSFJCIDUyIDQ5NwoKVGhpcyBlLW1haWwgbWF5IGNvbnRhaW4gY29uZmlk ZW50aWFsIGFuZC9vciBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLiBJZiB5b3UKYXJlIG5vdCB0aGUg aW50ZW5kZWQgcmVjaXBpZW50IChvciBoYXZlIHJlY2VpdmVkIHRoaXMgZS1tYWlsIGluIGVycm9y KQpwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXIgaW1tZWRpYXRlbHkgYW5kIGRlc3Ryb3kgdGhpcyBl LW1haWwuIEFueQp1bmF1dGhvcml6ZWQgY29weWluZywgZGlzY2xvc3VyZSBvciBkaXN0cmlidXRp b24gb2YgdGhlIG1hdGVyaWFsIGluIHRoaXMKZS1tYWlsIGlzIHN0cmljdGx5IGZvcmJpZGRlbi4K CmUtbWFpbHMgc2VudCBvdmVyIHRoZSBpbnRlcm5ldCBtYXkgaGF2ZSBiZWVuIHdyaXR0ZW4gdW5k ZXIgYSB3cm9uZyBuYW1lIG9yCmJlZW4gbWFuaXB1bGF0ZWQuIFRoYXQgaXMgd2h5IHRoaXMgbWVz c2FnZSBzZW50IGFzIGFuIGUtbWFpbCBpcyBub3QgYQpsZWdhbGx5IGJpbmRpbmcgZGVjbGFyYXRp b24gb2YgaW50ZW50aW9uLgoKQ29sbG9naWEKVW50ZXJuZWhtZW5zYmVyYXR1bmcgQUcKVWJpZXJy aW5nIDExCkQtNTA2NzggS8O2bG4KCmV4ZWN1dGl2ZSBib2FyZDoKS2FkaXIgQWtpbgpEci4gTWlj aGFlbCBIw7ZobmVyYmFjaAoKUHJlc2lkZW50IG9mIHRoZSBzdXBlcnZpc29yeSBib2FyZDoKSGFu cyBLcmlzdGlhbiBMYW5ndmEKClJlZ2lzdHJ5IG9mZmljZTogZGlzdHJpY3QgY291cnQgQ29sb2du ZQpSZWdpc3RlciBudW1iZXI6IEhSQiA1MiA0OTcKCioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioKCi0tLS0t LT1fTmV4dFBhcnRUTS0wMDAtOWMyNzNmMWMtYTJkZC00YWJhLWFiOTAtNGFhMjRiNWVmMjI1LS0K Cg== --===============1961673493606483215==--