<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Wed, Feb 7, 2018 at 7:09 PM Nicolas Ecarnot <<a href="mailto:nicolas@ecarnot.net">nicolas@ecarnot.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
TL; DR : qcow2 images keep getting corrupted. Any workaround?<br>
<br>
Long version:<br>
This discussion has already been launched by me on the oVirt and on<br>
qemu-block mailing list, under similar circumstances but I learned<br>
further things since months and here are some informations :<br>
<br>
- We are using 2 oVirt 3.6.7.5-1.el7.centos datacenters, using CentOS<br>
7.{2,3} hosts<br>
- Hosts :<br>
- CentOS 7.2 1511 :<br>
- Kernel = 3.10.0 327<br>
- KVM : 2.3.0-31<br>
- libvirt : 1.2.17<br>
- vdsm : 4.17.32-1<br>
- CentOS 7.3 1611 :<br>
- Kernel 3.10.0 514<br>
- KVM : 2.3.0-31<br>
- libvirt 2.0.0-10<br>
- vdsm : 4.17.32-1<br>
- Our storage is 2 Equallogic SANs connected via iSCSI on a dedicated<br>
network<br></blockquote><div><br></div><div>In 3.6 and iSCSI storage you have the issue of lvmetad service, activating </div><div>oVirt volumes by default, and also activating guest lvs inside oVirt raw volumes.</div><div>This can lead to data corruption if an lv was activated before it was extended</div><div>on another host, and the lv size on the host does not reflect the actual lv size.</div><div>We had many bugs related to this, check this for related bugs:</div><div><a href="https://bugzilla.redhat.com/1374545">https://bugzilla.redhat.com/1374545</a><br></div><div><br></div><div>To avoid this issue, you need to</div><div><br></div><div>1. edit /etc/lvm/lvm.conf global/use_lvmetad to:</div><div><br></div><div>use_lvmetad = 0</div><div><br></div><div>2. disable and mask these services:</div><div><br></div><div>- lvm2-lvmetad.socket</div><div>- lvm2-lvmetad.service</div><div><br></div><div>Note that this will may cause warnings from systemd during boot, the warnings</div><div>are harmless:</div><div><a href="https://bugzilla.redhat.com/1462792">https://bugzilla.redhat.com/1462792</a><br></div><div><br></div><div>For extra safety and better performance, you should also setup lvm filter on all hosts.</div><div><br></div><div>Check this for example how it is done in 4.x:</div><div><a href="https://www.ovirt.org/blog/2017/12/lvm-configuration-the-easy-way/">https://www.ovirt.org/blog/2017/12/lvm-configuration-the-easy-way/</a><br></div><div><br></div><div>Since you run 3.6 you will have to setup the filter manually in the same way.</div><div><br></div><div>Nir</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- Depends on weeks, but all in all, there are around 32 hosts, 8 storage<br>
domains and for various reasons, very few VMs (less than 200).<br>
- One peculiar point is that most of our VMs are provided an additional<br>
dedicated network interface that is iSCSI-connected to some volumes of<br>
our SAN - these volumes not being part of the oVirt setup. That could<br>
lead to a lot of additional iSCSI traffic.<br>
<br>
From times to times, a random VM appears paused by oVirt.<br>
Digging into the oVirt engine logs, then into the host vdsm logs, it<br>
appears that the host considers the qcow2 image as corrupted.<br>
Along what I consider as a conservative behavior, vdsm stops any<br>
interaction with this image and marks it as paused.<br>
Any try to unpause it leads to the same conservative pause.<br>
<br>
After having found (<a href="https://access.redhat.com/solutions/1173623" rel="noreferrer" target="_blank">https://access.redhat.com/solutions/1173623</a>) the<br>
right logical volume hosting the qcow2 image, I can run qemu-img check<br>
on it.<br>
- On 80% of my VMs, I find no errors.<br>
- On 15% of them, I find Leaked cluster errors that I can correct using<br>
"qemu-img check -r all"<br>
- On 5% of them, I find Leaked clusters errors and further fatal errors,<br>
which can not be corrected with qemu-img.<br>
In rare cases, qemu-img can correct them, but destroys large parts of<br>
the image (becomes unusable), and on other cases it can not correct them<br>
at all.<br>
<br>
Months ago, I already sent a similar message but the error message was<br>
about No space left on device<br>
(<a href="https://www.mail-archive.com/qemu-block@gnu.org/msg00110.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/qemu-block@gnu.org/msg00110.html</a>).<br>
<br>
This time, I don't have this message about space, but only corruption.<br>
<br>
I kept reading and found a similar discussion in the Proxmox group :<br>
<a href="https://lists.ovirt.org/pipermail/users/2018-February/086750.html" rel="noreferrer" target="_blank">https://lists.ovirt.org/pipermail/users/2018-February/086750.html</a><br>
<br>
<a href="https://forum.proxmox.com/threads/qcow2-corruption-after-snapshot-or-heavy-disk-i-o.32865/page-2" rel="noreferrer" target="_blank">https://forum.proxmox.com/threads/qcow2-corruption-after-snapshot-or-heavy-disk-i-o.32865/page-2</a><br>
<br>
What I read similar to my case is :<br>
- usage of qcow2<br>
- heavy disk I/O<br>
- using the virtio-blk driver<br>
<br>
In the proxmox thread, they tend to say that using virtio-scsi is the<br>
solution. Having asked this question to oVirt experts<br>
(<a href="https://lists.ovirt.org/pipermail/users/2018-February/086753.html" rel="noreferrer" target="_blank">https://lists.ovirt.org/pipermail/users/2018-February/086753.html</a>) but<br>
it's not clear the driver is to blame.<br>
<br>
I agree with the answer Yaniv Kaul gave to me, saying I have to properly<br>
report the issue, so I'm longing to know which peculiar information I<br>
can give you now.<br>
<br>
As you can imagine, all this setup is in production, and for most of the<br>
VMs, I can not "play" with them. Moreover, we launched a campaign of<br>
nightly stopping every VM, qemu-img check them one by one, then boot.<br>
So it might take some time before I find another corrupted image.<br>
(which I'll preciously store for debug)<br>
<br>
Other informations : We very rarely do snapshots, but I'm close to<br>
imagine that automated migrations of VMs could trigger similar behaviors<br>
on qcow2 images.<br>
<br>
Last point about the versions we use : yes that's old, yes we're<br>
planning to upgrade, but we don't know when.<br>
<br>
Regards,<br>
<br>
--<br>
Nicolas ECARNOT<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
</blockquote></div></div>