11:04 p.m.
Hi Nir,
Thank you for your time.
On 9/27/19 4:27 PM, Nir Soffer wrote:
On Fri, Sep 27, 2019, 12:37 Rik Theys <Rik.Theys(a)esat.kuleuven.be
<mailto:Rik.Theys@esat.kuleuven.be>> wrote:
Hi,
After upgrading to 4.3.6, my storage domain can no longer be
activated, rendering my data center useless.
My storage domain is local storage on a filesystem backed by
VDO/LVM. It seems 4.3.6 has added support for 4k storage.
My VDO does not have the 'emulate512' flag set.
This configuration is not supported before 4.3.6. Various operations
may fail when
reading or writing to storage.
I was not aware of this when I set it up as I did
not expect this to
influence a setup where oVirt uses local storage (a file system location).
4.3.6 detects storageblock size, creates compatible storage domain
metadata, and
consider the block size when accessing storage.
I've tried downgrading all packages on the host to the previous
versions (with ioprocess 1.2), but this does not seem to make any
difference.
Downgrading should solve your issue, but without any logs we only guess.
I was able to work around my issue by downgrading to ioprocess 1.1 (and
vdsm-4.30.24). Downgrading to only 1.2 did not solve my issue. With
ioprocess downgraded to 1.1, I did not have to downgrade the engine
(still on 4.3.6).
I think I now have a better understanding what happened that triggered this.
During a nightly yum-cron, the ioprocess and vdsm packages on the host
were upgraded to 1.3 and vdsm 4.30.33. At this point, the engine log
started to log:
2019-09-27 03:40:27,472+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-384418) [695f38cc] Executing with
domain map: {6bdf1a0d-274b-4195-8f
f5-a5c002ea1a77=active}
2019-09-27 03:40:27,646+02 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-384418) [695f38cc] Unexpected
return value: Status [code=348, message=Block size does not match
storage block size: 'block_size=512, storage_block_size=4096']
2019-09-27 03:40:27,646+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-384418) [695f38cc] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 483c7a17
I did not notice at first that this was a storage related issue and
assumed it may get resolved by also upgrading the engine. So in the
morning I upgraded the engine to 4.3.6 but this did not resolve my issue.
I then found the above error in the engine log. In the release notes of
4.3.6 I read about the 4k support.
I then downgraded ioprocess (and vdsm) to ioprocess 1.2 but that did
also not solve my issue. This is when I contacted the list with my question.
Afterwards I found in the ioprocess rpm changelog that (partial?) 4k
support was also in 1.2. I kept on downgrading until I got ioprocess 1.1
(without 4k support) and at this point I could re-attach my storage domain.
You mention above that 4.3.6 will detect the block size and configure
the metadata on the storage domain? I've checked the dom_md/metadata
file and it shows:
ALIGNMENT=1048576
*BLOCK_SIZE=512*
CLASS=Data
DESCRIPTION=studvirt1-Local
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=1
POOL_DESCRIPTION=studvirt1-Local
POOL_DOMAINS=6bdf1a0d-274b-4195-8ff5-a5c002ea1a77:Active
POOL_SPM_ID=-1
POOL_SPM_LVER=-1
POOL_UUID=085f02e8-c3b4-4cef-a35c-e357a86eec0c
REMOTE_PATH=/data/images
ROLE=Master
SDUUID=6bdf1a0d-274b-4195-8ff5-a5c002ea1a77
TYPE=LOCALFS
VERSION=5
_SHA_CKSUM=9dde06bbc9f2316efc141565738ff32037b1ff66
I assume that at this point it works because ioprocess 1.1 does not
report the block size to the engine (as it doesn't support this option?)?
Can I update the storage domain metadata manually to report 4096 instead?
I also noticed that the storage_domain_static table has the block_size
stored. Should I update this field at the same time as I update the
metadata file?
If the engine log and database dump is still needed to better understand
the issue, I will send it on Monday.
Regards,
Rik
Should I also downgrade the engine to 4.3.5 to get this to work
again. I expected the downgrade of the host to be sufficient.
As an alternative I guess I could enable the emulate512 flag on
VDO but I can not find how to do this on an existing VDO volume.
Is this possible?
Please share more data so we can understand the failure:
- complete vdsm log showing the failure to activate the domain
- with 4.3.6
- with 4.3.5 (after you downgraded
- contents of
/rhev/data-center/mnt/_<domaindir>/domain-uuid/dom_md/metadata
(assuming your local domain mount is /domaindir)
- engine db dump
Nir
Regards,
Rik
On 9/26/19 4:58 PM, Sandro Bonazzola wrote:
>
> The oVirt Project is pleased to announce the general availability
> of oVirt 4.3.6 as of September 26th, 2019.
>
> This update is the sixth in a series of stabilization updates to
> the 4.3 series.
>
> This release is available now on x86_64 architecture for:
>
> * Red Hat Enterprise Linux 7.7 or later (but < 8)
>
> * CentOS Linux (or similar) 7.7 or later (but < 8)
>
> This release supports Hypervisor Hosts on x86_64 and ppc64le
> architectures for:
>
> * Red Hat Enterprise Linux 7.7 or later (but < 8)
>
> * CentOS Linux (or similar) 7.7 or later (but < 8)
>
> * oVirt Node 4.3 (available for x86_64 only)
>
> Due to Fedora 28 being now at end of life this release is missing
> experimental tech preview for x86_64 and s390x architectures for
> Fedora 28.
>
> We are working on Fedora 29 and 30 support and we may
> re-introduce experimental support for Fedora in next release.
>
> See the release notes [1] for installation / upgrade instructions
> and a list of new features and bugs fixed.
>
> Notes:
>
> - oVirt Appliance is already available
>
> - oVirt Node is already available[2]
>
>
> oVirt Node and Appliance have been updated including:
>
> - oVirt4.3.6: http://www.ovirt.org/release/4.3.6/
>
> - Wildfly17.0.1:
> https://wildfly.org/news/2019/07/07/WildFly-1701-Released/
>
> - Latest CentOS 7.7updates including:
>
> *
>
> Release for CentOS Linux 7 (1908) on the x86_64 Architecture
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023405....
>
> *
>
> CEBA-2019:2601 CentOS 7 NetworkManager BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023423....
>
> *
>
> CEBA-2019:2023 CentOS 7 efivar BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023445....
>
> *
>
> CEBA-2019:2614 CentOS 7 firewalld BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023412....
>
> *
>
> CEBA-2019:2227 CentOS 7 grubby BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023441....
>
> *
>
> CESA-2019:2258 Moderate CentOS 7 http-parser Security Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023439....
>
> *
>
> CESA-2019:2600 Important CentOS 7 kernel Security Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023444....
>
> *
>
> CEBA-2019:2599 CentOS 7 krb5 BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023420....
>
> *
>
> CEBA-2019:2358 CentOS 7 libguestfs BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023421....
>
> *
>
> CEBA-2019:2679 CentOS 7 libvirt BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023422....
>
> *
>
> CEBA-2019:2501 CentOS 7 rsyslog BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023431....
>
> *
>
> CEBA-2019:2355 CentOS 7 selinux-policy BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023432....
>
> *
>
> CEBA-2019:2612 CentOS 7 sg3_utils BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023433....
>
> *
>
> CEBA-2019:2602 CentOS 7 sos BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023434....
>
> *
>
> CEBA-2019:2564 CentOS 7 subscription-manager BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023435....
>
> *
>
> CEBA-2019:2356 CentOS 7 systemd BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023436....
>
> *
>
> CEBA-2019:2605 CentOS 7 tuned BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023437....
>
> *
>
> CEBA-2019:2871 CentOS 7 tzdata BugFix Update
>
<https://lists.centos.org/pipermail/centos-announce/2019-September/023450....
>
> - latest CentOS Virt and Storage SIG updates:
>
> *
>
> Ansible2.8.5:
>
https://github.com/ansible/ansible/blob/stable-2.8/changelogs/CHANGELOG-v...
>
> *
>
> Glusterfs6.5:
> https://docs.gluster.org/en/latest/release-notes/6.5/
>
> *
>
> QEMU KVM EV2.12.0-33.1 :
> https://cbs.centos.org/koji/buildinfo?buildID=26484
>
>
> Given the amount of security fixes provided by this release,
> upgrade is recommended as soon as practical.
>
>
> Additional Resources:
>
> * Read more about the oVirt 4.3.6 release
> highlights:http://www.ovirt.org/release/4.3.6/
>
> * Get more oVirt Project updates on Twitter:
> https://twitter.com/ovirt
>
> * Check out the latest project news on the oVirt
> blog:http://www.ovirt.org/blog/
>
> [1] http://www.ovirt.org/release/4.3.6/
>
> [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com <mailto:sbonazzo@redhat.com>
>
> <https://www.redhat.com/>
>
> *Red Hat respects your work life balance. Therefore there is no
> need to answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
> _______________________________________________
> Users mailing list --users(a)ovirt.org <mailto:users@ovirt.org>
> To unsubscribe send an email tousers-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
> Privacy Statement:https://www.ovirt.org/site/privacy-policy/
> oVirt Code of
Conduct:https://www.ovirt.org/community/about/community-guidelines/
> List
Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/AY...
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
_______________________________________________
Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JPIYWV2OUNL...
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>