- Users - oVirt List Archives

4.3.5 update error
by Kapetanakis Giannis 06 Aug '19

06 Aug '19

Hi, Trying today engine-setup to update from 4.3.3 -> 4.3.5 I got this error: [ ERROR ] Failed to execute stage 'Setup validation': local variable 'snapshot_cl' referenced before assignment In log file I see: 2019-07-31 10:43:00,375+0300 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 437, in _validation self._checkSnapshotCompatibilityVersion() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 212, in _checkSnapshotCompatibilityVersion 'v': snapshot_cl, UnboundLocalError: local variable 'snapshot_cl' referenced before assignment 2019-07-31 10:43:00,381+0300 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Setup validation': local variable 'snapshot_cl' referenced before assignment Any hints? thanks Giannis

5 5

[ANN] oVirt Engine 4.3.5 async update is now available
by Sandro Bonazzola 06 Aug '19

06 Aug '19

The oVirt Team has just released a new version of ovirt-engine package that fixes one upgrade related issue.[1] We recommend to users experiencing upgrade issues to try again with this new release. Thanks, [1] https://bugzilla.redhat.com/1734699 -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

oVirt tools
by old.good.mail＠gmail.com 06 Aug '19

06 Aug '19

Greetings, After installing oVirt tools on my w2016 virtual server I've noticed that all devices (nic, disks, etc.) are connected as USB devices and I have option available to eject those devices. Is there any way to prvent those devices being ejected on behalf of VM user? (e.g. make sure noone inside the VM can actually eject nic, disk or something)

4 4

[ANN] oVirt 4.3.5 second async is now available
by Sandro Bonazzola 05 Aug '19

05 Aug '19

The oVirt Team has just released a new version of the following packages: - ovirt-ansible-hosted-engine-setup-1.0.26 - ovirt-release43-4.3.5.2 The async release includes additional fixes: - Bug 1734171 <https://bugzilla.redhat.com/show_bug.cgi?id=1734171> - RHV-M 4.3.4 does not boot with fips=1 when OpenSCAP profile used resulting in non FIPS compliant certificates Notes: - oVirt Node is already available[2] oVirt Node has been updated including above packages and: - Latest CentOS Virt SIG update: - CESA-2019:1968 - Important qemu-kvm-ev Security Update Upstream details at https://access.redhat.com/errata/RHSA-2019:1968 - Latest CentOS 7.6 updates including: - CEBA-2019:1899 CentOS 7 bind BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023376.html> - CESA-2019:1880 Low CentOS 7 curl Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023377.html> - CEBA-2019:1889 CentOS 7 ipa BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023380.html> - CESA-2019:1873 Important CentOS 7 kernel Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023374.html> - CEBA-2019:1895 CentOS 7 systemd BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023393.html> - CEBA-2019:1875 CentOS 7 samba BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023397.html> - CESA-2019:1884 Moderate CentOS 7 libssh2 Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023383.html> - CEBA-2019:1874 CentOS 7 net-snmp BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023386.html> - CEBA-2019:1878 CentOS 7 selinux-policy BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023391.html> - CEBA-2019:1885 CentOS 7 sos BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023392.html> - CEEA-2019:1887 CentOS 7 sysstat Enhancement Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023396.html> - CEBA-2019:1901 CentOS 7 tuned BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023394.html> Additional Resources: * Read more about the oVirt 4.3.5 release highlights: http://www.ovirt.org/release/4.3.5/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.5/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Linux VM keeps crashing on copying files "with lost communication with qemu"
by kevin.doyle＠manchester.ac.uk 02 Aug '19

02 Aug '19

HI I think it is a bug with qemu, and isci disks. Has anyone else seen this issue VM crashes and lost communication with qemu Can I update qemu ? if so what is the latest support version to use on Ovirt 4.3.2.1-1.el7 Current versions of qemu installed Installed Packages qemu-img-ev.x86_64 10:2.12.0-18.el7_6.3.1 qemu-kvm-common-ev.x86_64 10:2.12.0-18.el7_6.3.1 qemu-kvm-ev.x86_64 10:2.12.0-18.el7_6.3.1 Many thanks Kevin

2 1

Use public-signed SSL certs?
by Chris Adams 02 Aug '19

02 Aug '19

I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work. I'm trying to avoid ever needing to install a special CA cert in browsers. -- Chris Adams <cma(a)cmadams.net>

6 19

Re: Manual Migration not working and Dashboard broken after 4.3.4 update
by Strahil 02 Aug '19

02 Aug '19

I'm not sure, but I always thought that you need an agent for live migrations. You can always try installing either qemu-guest-agent or ovirt-guest-agent and check if live migration between hosts is possible. Have you set the new cluster/dc version ? Best Regards Strahil NikolovOn Jul 9, 2019 17:42, Neil <nwilson123(a)gmail.com> wrote: > > I remember seeing the bug earlier but because it was closed thought it was unrelated, this appears to be it.... > > https://bugzilla.redhat.com/show_bug.cgi?id=1670701 > > Perhaps I'm not understanding your question about the VM guest agent, but I don't have any guest agent currently installed on the VM, not sure if the output of my qemu-kvm process maybe answers this question?.... > > /usr/libexec/qemu-kvm -name guest=Headoffice.cbl-ho.local,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Headoffice.cbl-ho.lo/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on,rtm=on,hle=on -m 8192 -realtime mlock=off -smp 8,maxcpus=64,sockets=16,cores=4,threads=1 -numa node,nodeid=0,cpus=0-7,mem=8192 -uuid 9a6561b8-5702-43dc-9e92-1dc5dfed4eef -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=4C4C4544-0034-5810-8033-C2C04F4E4B32,uuid=9a6561b8-5702-43dc-9e92-1dc5dfed4eef -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2019-07-09T10:26:53,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/59831b91-00a5-01e4-0294-000000000018/8a607f8a-542a-473c-bb18-25c05fe2a3d4/images/56e8240c-a172-4f52-b0c1-2bddc4f34f93/9f245467-d31d-4f5a-8037-7c5012a4aa84,format=qcow2,if=none,id=drive-virtio-disk0,serial=56e8240c-a172-4f52-b0c1-2bddc4f34f93,werror=stop,rerror=stop,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:5b,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,fd=35,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,fd=36,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.0.1.11,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=8388608,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on > > Please shout if you need further info. > > Thanks. > > > > > > > On Tue, Jul 9, 2019 at 4:17 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Shouldn't cause that problem. >> >> You have to find the bug in bugzilla and report a regression (if it's not closed) , or open a new one and report the regression. >> As far as I remember , only the dashboard was affected due to new features about vdo disk savings. >>

5 13

no network interface
by A S 01 Aug '19

01 Aug '19

Hi. I had ovirt running with a VM but it suddenly broke. I wiped it all and did a reinstall but now I am not able to connect my host to the network. the host is always saying status='unassigned'. in the network page where I would drag and drop a connection, there is no interface to connect to. its this page https://ovirt.org/images/wiki/SetupNetworksNew.png?1478101462 but there is nothing in the interfaces column on the left Can anyone point me in the right direction to fix this? Thanks I have only one machine with the engine and host on it.

3 2

[ANN] oVirt 4.3.6 First Release Candidate is now available for testing
by Sandro Bonazzola 01 Aug '19

01 Aug '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.6 First Release Candidate for testing, as of August 1st, 2019. This update is a release candidate of the sixth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later (but <8) * CentOS Linux (or similar) 7.6 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later (but <8) * CentOS Linux (or similar) 7.6 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is laready available - oVirt Node is already available[2] Additional Resources: * Read more about the oVirt 4.3.6 release highlights: http://www.ovirt.org/release/4.3.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.6/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

[ANN] oVirt 4.3.5 first async is now available
by Sandro Bonazzola 31 Jul '19

31 Jul '19

The oVirt Team has just released a new version of the following packages: - ovirt-ansible-disaster-recovery-1.2.0 - ovirt-ansible-hosted-engine-setup-1.0.24 - ovirt-engine-dwh-4.3.5.1 - ovirt-engine-metrics-1.3.3.3 - ovirt-release43-4.3.5.1 The async release includes additional fixes: - [BZ 1721073 <https://bugzilla.redhat.com/1721073>] Run metrics ansible role only if the ovirt-engine-metrics rpm installed - [BZ 1731871 <https://bugzilla.redhat.com/1731871>] Unable to install metrics VM on rhev - failed to connect to master VM - [BZ 1727064 <https://bugzilla.redhat.com/1727064>] Deployment from Bastion Machine fails with Ansible 2.8 on upstream - [BZ 1698888 <https://bugzilla.redhat.com/1698888>] Update task "Get bastion ssh public key" to use ansible module other then command - [BZ 1730384 <https://bugzilla.redhat.com/1730384>] Connection closed with an exception - [BZ 1683157 <https://bugzilla.redhat.com/1683157>] CVE-2019-10194 <https://access.redhat.com/security/cve/cve-2019-10194> ovirt-engine-metrics: disclosure of sensitive passwords in log files and ansible playbooks Notes: - oVirt Appliance is already available - oVirt Node is already available[2] oVirt Node and Appliance have been updated including above packages and: - latest CentOS Virt and Storage SIG updates: - ansible-2.8.3: https://github.com/ansible/ansible/blob/stable-2.8/changelogs/CHANGELOG-v2.… - glusterfs-6.4: https://docs.gluster.org/en/latest/release-notes/6.4 - cockpit 195: https://cockpit-project.org/blog/cockpit-195.html - openvswitch 2.11: http://openvswitch.org/releases/NEWS-2.11.0 Additional Resources: * Read more about the oVirt 4.3.5 release highlights: http://www.ovirt.org/release/4.3.5/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.5/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

oVirt 4.3.5 is now generally available
by Sandro Bonazzola 30 Jul '19

30 Jul '19

The oVirt Project is pleased to announce the general availability of oVirt 4.3.5 as of July 30th, 2019. This update is the fourth in a series of stabilization updates to the 4.3 series. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later (but < 8) * CentOS Linux (or similar) 7.6 or later (but < 8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later (but < 8) * CentOS Linux (or similar) 7.6 or later (but < 8) * oVirt Node 4.3 (available for x86_64 only) Due to Fedora 28 being now at end of life this release is missing experimental tech preview for x86_64 and s390x architectures for Fedora 28. We are working on Fedora 29 and 30 support and we may re-introduce experimental support for Fedora in next release. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available[2] Please note that released Appliance and Node are missing ansible-2.8.3 and Gluster 6.4. We already tested this release with these versions but due to an issue on CentOS signing server we didn’t got he builds on CentOS mirrors. Since a new RHEL batch update has been shipped last night we’ll respin once all the new packages will be available on CentOS mirrors. oVirt Node and Appliance have been updated including: - oVirt 4.3.5: http://www.ovirt.org/release/4.3.5/ - wildfly-17.0.0: https://wildfly.org/news/2019/06/10/WildFly17-Final-Released/ - Latest CentOS updates including: - CESA-2019:1294 Important CentOS 7 bind Security Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023321.html> - CEBA-2019:1331 CentOS 7 glib2 BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023325.html> - CEBA-2019:1340 CentOS 7 lvm2 BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023354.html> - CEBA-2019:1338 CentOS 7 glibc BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023352.html> - CESA-2019:1815 Moderate CentOS 7 java-1.8.0-openjdk Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023373.html> - CEBA-2019:1339 CentOS 7 kernel BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023326.html> - CEBA-2019:1333 CentOS 7 libteam BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023353.html> - CEEA-2019:1612 CentOS 7 microcode_ctl Enhancement Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023344.html> - CESA-2019:1587 Important CentOS 7 python Security Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023337.html> - CEBA-2019:1336 CentOS 7 systemd BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023356.html> - CEBA-2019:1703 CentOS 7 tzdata BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023363.html> - CESA-2019:1619 Important CentOS 7 vim Security Update <https://lists.centos.org/pipermail/centos-announce/2019-July/023345.html> - CEBA-2019:1345 CentOS 7 fence-agents BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023323.html> - CESA-2019:1579 Important CentOS 7 libvirt Security Update <https://lists.centos.org/pipermail/centos-announce/2019-June/023336.html> - - latest CentOS Virt and Storage SIG updates: - ansible-2.8.2: https://github.com/ansible/ansible/blob/stable-2.8/changelogs/CHANGELOG-v2.… - glusterfs-6.3: https://docs.gluster.org/en/latest/release-notes/6.3 - cockpit 195: https://cockpit-project.org/blog/cockpit-195.html Additional Resources: * Read more about the oVirt 4.3.5 release highlights: http://www.ovirt.org/release/4.3.5/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.5/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*

1 0

Leeroy Jenkins cloud plugin for oVirt
by Vrgotic, Marko 30 Jul '19

30 Jul '19

Hey oVirt, How are you all doing? Has anyone used or has experience with Leeroy Jenkins on oVIrt. We need to move master ci and its slaves to oVirt, but cannot find suitable plugin for it. For builds we run, Jenkins need to spawn slaves, for which plugin is required. Please help if possible. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

1 0

Trouble initializing new VM in 4.2.8.2-1.el7
by David Johnson 30 Jul '19

30 Jul '19

Good evening, Thanks in advance, I'm trying to set up a new VM in my cluster, and things appear to have hung up initializing the 80GB boot partition for a new Windows VM. The system has been sitting like this for 6 hours. What am I looking for to resolve this issue? Can I kill this process safely and start over? [image: image.png] [image: image.png] Regards, David Johnson Director of Development, Maxis Technology 844.696.2947 ext 702 (o) | 479.531.3590 (c) djohnson(a)maxistechnology.com [image: Maxis Techncology] <http://www.maxistechnology.com> www.maxistechnology.com *stay connected <http://www.linkedin.com/in/pojoguy>*

2 1

Moving data to new storage appliance
by David Johnson 30 Jul '19

30 Jul '19

Hi everyone, I'm sorry to bother y'all with another noob question. We are in the process of retiring the old storage appliance that backed our oVirt cluster in favor of a new appliance. I have migrated all of the active VM storage to the new appliance, but can't see how to migrate the "base versions". My understanding is that if I just drop the storage with the base versions then the derived VM's will cease to be functional. Please advise.

2 3

bond for vm interfaces
by Edoardo Mazza 29 Jul '19

29 Jul '19

Hello everyone, I need to create a bond for vm interfaces but I don't kown what is the best solution, you can help me? Thanks Edoardo

5 4

Re: Active-Passive DR: mutual for different storage domains possible?
by Gianluca Cecchi 29 Jul '19

29 Jul '19

On Thu, Jul 25, 2019 at 2:21 PM Eyal Shenitzky <eshenitz(a)redhat.com> wrote: > On Thu, Jul 25, 2019 at 3:02 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com> > wrote: > >> On Thu, Jul 25, 2019 at 1:54 PM Eyal Shenitzky <eshenitz(a)redhat.com> >> wrote: >> >>> >>> Please notice that a automation python scripts created in order to >>> facilitate the DR process. >>> You can find them under - path/to/your/dr/folder/files. >>> >>> You can use those scripts to generate the mapping, test the generated >>> mapping and start the failover/failback. >>> >>> I strongly recommend to use it. >>> >>> >> Yes, I have used it to create the disaster_recovery_vars.yml mapping file >> and then populating it with the secondary site information, thanks. >> My doubt was about any difference in playbook actions between "failover" >> (3.3) and "discreet failover test" (B.1), as the executed playbook and >> tags are the same. >> > > No, the only difference is that you disable the storage replication by > yourself, this way you can test the failover while the other "primary" site > is still active. > > First "discreet failover test" was a success!!! Great. Storage domain attached, templates imported and the only VM defined at source correctly started (at source I configured link down for the VM, inherited at target, so no collisions). Elapsed between beginning of ovirt connection, until first template import has been about 6 minutes. ... Template TOL76 has been successfully imported from the given configuration. 7/25/19 3:26:58 PM Storage Domain ovsd3910 was attached to Data Center SVIZ3-DR by admin@internal-authz 7/25/19 3:26:46 PM Storage Domains were attached to Data Center SVIZ3-DR by admin@internal-authz 7/25/19 3:26:46 PM Storage Domain ovsd3910 (Data Center SVIZ3-DR) was activated by admin@internal-authz 7/25/19 3:26:46 PM ... Storage Pool Manager runs on Host ovh201. (Address: ovh201.), Data Center SVIZ3-DR. 7/25/19 3:26:36 PM Data Center is being initialized, please wait for initialization to complete. 7/25/19 3:23:53 PM Storage Domain ovsd3910 was added by admin@internal-authz 7/25/19 3:20:43 PM Disk Profile ovsd3910 was successfully added (User: admin@internal-authz). 7/25/19 3:20:42 PM User admin@internal-authz connecting from '10.4.192.43' using session 'xxx' logged in. 7/25/19 3:20:35 PM Some notes: 1) iSCSI multipath my storage domains are iSCSI based and my hosts have two network cards to reach the storage. I'm using EQL that doesn't support bonding and has one portal that all initiators use. So in my primary env I configured "iSCSI Multipathing" tab in Compute --> Datacenter --> Datacenter_Name window. But this tab appears only when you activate the storage. So during the ansible playbook run the iSCSI connection has been activated through the "default" iscsi interface I can then: - configure "iSCSI Multipathing" - shutdown VM - put host into maintenance - remove the default iSCSI session that has not been removed on host iscsiadm -m session -r 6 -u - activate host now I have: [root@ov201 ~]# iscsiadm -m session tcp: [10] 10.10.100.8:3260,1 iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910 (non-flash) tcp: [9] 10.10.100.8:3260,1 iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910 (non-flash) [root@ov201 ~]# with # multipath -l 364817197c52fd899acbe051e0377dd5b dm-29 EQLOGIC ,100E-00 size=1.0T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active |- 23:0:0:0 sdb 8:16 active undef running `- 24:0:0:0 sdc 8:32 active undef running - start vm The I do a cleanup: 1. Detach the storage domains from the secondary site. 2. Enable storage replication between the primary and secondary storage domains. The storage domain remains as "unattached" in DR environment Then I executed the test again and during connection I got this error about 40 seconds after run of playbook TASK [oVirt.disaster-recovery : Import iSCSI storage domain] *************************************** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. failed: [localhost] (item=iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910) => {"ansible_loop_var": "dr_target", "changed": false, "dr_target": "iqn.2001-05.com.equallogic:4-771816-99d82fc59-5bdd77031e05beac-ovsd3910", "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."} In webadmin gui of DR env I see: VDSM ov201 command CleanStorageDomainMetaDataVDS failed: Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" 7/25/19 4:50:43 PM What could be the cause of this? In vdsm.log: 2019-07-25 16:50:43,196+0200 INFO (jsonrpc/1) [vdsm.api] FINISH forcedDetachStorageDomain error=Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" from=::ffff:10.4.192.79,49038, flow_id=4bd330d1, task_id=c0dfac81-5c58-427d-a7d0-e8c695448d27 (api:52) 2019-07-25 16:50:43,196+0200 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='c0dfac81-5c58-427d-a7d0-e8c695448d27') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in forcedDetachStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 856, in forcedDetachStorageDomain self._detachStorageDomainFromOldPools(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 834, in _detachStorageDomainFromOldPools dom.acquireClusterLock(host_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 910, in acquireClusterLock self._manifest.acquireDomainLock(hostID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 527, in acquireDomainLock self._domainLock.acquire(hostID, self.getDomainLease()) File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 419, in acquire "Cannot acquire %s" % (lease,), str(e)) AcquireLockFailure: Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" 2019-07-25 16:50:43,196+0200 INFO (jsonrpc/1) [storage.TaskManager.Task] (Task='c0dfac81-5c58-427d-a7d0-e8c695448d27') aborting: Task is aborted: 'Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name=\'SDM\', path=\'/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases\', offset=1048576), err=(-243, \'Sanlock resource not acquired\', \'Lease is held by another host\')"' - code 651 (task:1181) 2019-07-25 16:50:43,197+0200 ERROR (jsonrpc/1) [storage.Dispatcher] FINISH forcedDetachStorageDomain error=Cannot obtain lock: "id=56eadc97-5731-40cf-8409-aff58d8ffd11, rc=-243, out=Cannot acquire Lease(name='SDM', path='/dev/56eadc97-5731-40cf-8409-aff58d8ffd11/leases', offset=1048576), err=(-243, 'Sanlock resource not acquired', 'Lease is held by another host')" (dispatcher:83) 2019-07-25 16:50:43,197+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.detach failed (error 651) in 24.12 seconds (__init__:312) 2019-07-25 16:50:44,180+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:10.4.192.79,49038 (api:48) 2019-07-25 16:50:44,222+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:10.4.192.79,49038, task_id=8a7a0302-4ee3-49a8-a3f7-f9636a123765 (api:48) 2019-07-25 16:50:44,222+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={} from=::ffff:10.4.192.79,49038, task_id=8a7a0302-4ee3-49a8-a3f7-f9636a123765 (api:54) 2019-07-25 16:50:44,223+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:10.4.192.79,49038, task_id=fb09923c-0888-4c3f-9b8a-a7750592da22 (api:48) 2019-07-25 16:50:44,223+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.4.192.79,49038, task_id=fb09923c-0888-4c3f-9b8a-a7750592da22 (api:54) After putting host into maintenance + reboot of the host and re-run of playbook, all went well again.. 2) Mac Address pools I notice that the imported VM has preserved the link state (down in my case) but not the mac address. The mac address is the one defined in my target engine, that is different from source engine to avoid overlap of mac adresses Is this an option I can customize? In general a VM could have problems when changing its mac address... 3) Clean up dest DC after "discreet failover test" The guide says: 1. Detach the storage domains from the secondary site. 2. Enable storage replication between the primary and secondary storage domains. better to also restart the DR hosts? 4) VM consistency Can we say that all the imported VMs will be "crash consistent"? Thanks, Gianluca

2 5

Re: engine-setup failure on 4.1 -> 4.2
by Strahil 27 Jul '19

27 Jul '19

Try to remove the maintenance, wait 10 min and then put it back: hosted-engine --set-maintenance --mode=none; sleep 600; hosted-engine --set-maintenance --mode=global ; sleep 300 Best Regards, Strahil NikolovOn Jul 26, 2019 20:49, Alex K <rightkicktech(a)gmail.com> wrote: > > I repeated the same upgrade steps to another cluster and although I was receiving same warnings about the db the upgrade completed successfully. > > Is there a way I can manually inform engine db about the maintenance status? I was thinking that in this way the engine would proceed with the remaining steps. > > On Thu, Jul 25, 2019 at 3:55 PM Alex K <rightkicktech(a)gmail.com> wrote: >> >> Hi all, >> >> I have a self hosted engine setup, with 3 servers. >> I have successfully upgraded several other installations, from 4.1 to 4.2. >> On one of them I am encountering and issue with the engine-setup. >> >> I get the following warning: >> >> Found the following problems in PostgreSQL configuration for the Engine database: >> It is required to be at least '8192' >> >> Please note the following required changes in postgresql.conf on 'localhost': >> 'work_mem' is currently '1024'. It is required to be at least '8192'. >> postgresql.conf is usually in /var/lib/pgsql/data, /var/opt/rh/rh-postgresql95/lib/pgsql/data, or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes. >> The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: >> >> Then, if I select Yes to proceed, I get: >> >> [WARNING] This release requires PostgreSQL server 9.5.14 but the engine database is currently hosted on PostgreSQL server 9.2.24. >> >> Then finally: >> [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. >> In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. >> >> [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. >> [ INFO ] Stage: Clean up >> Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190725124653-hvekp2.log >> [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190725125154-setup.conf' >> [ INFO ] Stage: Pre-termination >> [ INFO ] Stage: Termination >> [ ERROR ] Execution of setup failed >> >> I have put the cluster on global maintenance, though the engine thinks it is not. >> >> Are any steps that I may follow to avoid above? >> I am attaching also the last full setup log. >> Thank you! >> >> Alex >> >> >>

1 0

engine-setup failure on 4.1 -> 4.2
by Alex K 26 Jul '19

26 Jul '19

Hi all, I have a self hosted engine setup, with 3 servers. I have successfully upgraded several other installations, from 4.1 to 4.2. On one of them I am encountering and issue with the engine-setup. I get the following warning: Found the following problems in PostgreSQL configuration for the Engine database: It is required to be at least '8192' Please note the following required changes in postgresql.conf on 'localhost': 'work_mem' is currently '1024'. It is required to be at least '8192'. postgresql.conf is usually in /var/lib/pgsql/data, /var/opt/rh/rh-postgresql95/lib/pgsql/data, or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes. The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]: Then, if I select Yes to proceed, I get: [WARNING] This release requires PostgreSQL server 9.5.14 but the engine database is currently hosted on PostgreSQL server 9.2.24. Then finally: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. [ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set. [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20190725124653-hvekp2.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20190725125154-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed I have put the cluster on global maintenance, though the engine thinks it is not. Are any steps that I may follow to avoid above? I am attaching also the last full setup log. Thank you! Alex

1 1

Clone function seem broken - regarding disk virtual size
by Vrgotic, Marko 26 Jul '19

26 Jul '19

Dear oVirt, All our CentOS templates are “baked” with 40GB disk (sparse) QCOW. When Creating new VM from template, using Thin storage allocation: * VM is created with 40GB disk (reading from oVirt GUI) and * from OS fdisk and/or parted report 40GB disk size. * The / partition is also matching expected size When Creating VM from template, using default Clone storage allocation: * VM is created with 40GB disk (reading from oVIrt GUI) * However: * from OS fdsik and/or parted report only actual used space, which in our case is 8GB. * The / partition is matching only actual disk size, not virtual one Why is Clone not taking into account Virtual size of the Template disk? Is it intended behavior that when image is cloned from template, only actual disk size information is copied, not the virtual? Please assist. Kindly awaiting your reply. oVirt 4.3.4 with SHE and NFS based storage and Local. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic Sr. System Engineer @ System Administration m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> tel. +31 (0)35 677 4131 ActiveVideo BV Mediacentrum 3741 Joop van den Endeplein 1 1217 WJ Hilversum

1 0

Re: major network changes
by Strahil 25 Jul '19

25 Jul '19

The CA can be downloaded via the web , or you can tell curl to just ignore the engine's cert via the '-k' flag. It will show you if the health page is working. Best Regards, Strahil NikolovOn Jul 24, 2019 19:39, carl langlois <crl.langlois(a)gmail.com> wrote: > > Strahil, not sure what to put for the --cacert. > > Yes Derek your are right at one point the port 8702 stop listening. > > tcp6 0 0 127.0.0.1:8702 :::* LISTEN 1607/ovirt-engine > > After some time the line above disappear. I am trying to figure why this port is being close after some time when the engine is running on the host on the 248.x network. On the 236.x network this port is kept alive all the time. > If you have any hint on why this port is closing do not hesitate because i am starting to be out of ideas. :-) > > > Thanks & Regards > > Carl > > > > > > > On Wed, Jul 24, 2019 at 11:11 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> A healthy engine should report: >> [root@ovirt1 ~]# curl --cacert CA https://engine.localdomain/ovirt-engine/services/health;echo >> DB Up!Welcome to Health Status! >> >> Of course you can use the '-k' switch to verify the situation. >> >> Best Regards, >> Strahil Nikolov >> >> В сряда, 24 юли 2019 г., 17:43:59 ч. Гринуич+3, Derek Atkins <derek(a)ihtfp.com> написа: >> >> >> Hi, >> >> carl langlois <crl.langlois(a)gmail.com> writes: >> >> > If i try to access http://ovengine/ovirt-engine/services/health >> > i always get "Service Unavailable" in the browser and each time i it reload in >> > the browser i get in the error_log >> > >> > [proxy_ajp:error] [pid 1868] [client 10.8.1.76:63512] AH00896: failed to make >> > connection to backend: 127.0.0.1 >> > [Tue Jul 23 14:04:10.074023 2019] [proxy:error] [pid 1416] (111)Connection >> > refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed >> >> Sounds like a service isn't running on port 8702. >> >> > Thanks & Regards >> > >> > Carl >> >> -derek >> >> -- >> Derek Atkins 617-623-3745 >> >> derek(a)ihtfp.com >> www.ihtfp.com >> Computer and Internet Security Consultant

3 3

Hosted Engine VM - CPU and Memory Sizing
by paul.christian.suba＠cevalogistics.com 25 Jul '19

25 Jul '19

Hi, Is there a recommended CPU and memory size for the Hosted Engine VM? We have what started as a 4 node physical cluster lab with 4 vms that has now grown to 44vms. The dashboard is slow to load information and the HE VM is consistently seen with 99% CPU with the breakdown below. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12076 postgres 20 0 506516 276172 143772 R 99.3 1.7 206:17.15 postmaster 10337 postgres 20 0 484756 254156 144000 R 99.0 1.6 179:52.35 postmaster 38603 postgres 20 0 500068 267836 143992 S 70.1 1.6 528:04.13 postmaster 49217 postgres 20 0 468736 235484 143624 S 19.3 1.4 41:57.55 postmaster 5569 ovirt 20 0 6430912 2.3g 6368 S 1.3 14.5 894:20.75 java We used the default 4 CPU and 16 GB RAM. This is oVirt 4.3.1. I am curious to find out as well if the postgres processes using 99% CPU is normal?

2 1

ovirt-engine-appliance ova
by jingjie.jiang＠oracle.com 25 Jul '19

25 Jul '19

Hi, Can someone tell me how to generate ovirt-engine-appliance ova file in ovirt-engine-appliance-4.3-20190610.1.el7.x86_64.rpm? I tried to import ovirt-engine-appliance ova(ovirt-engine-appliance-4.3-20190610.1.el7.ova) from ovirt-engine, but I got error as following: Failed to load VM configuration from OVA file: /var/tmp/ovirt-engine-appliance-4.2-20190121.1.el7.ova I guess ovirt-engine-appliance-4.2-20190121.1.el7.ova has more than CentOS7.6. Thanks, Jingjie

5 7

Re: major network changes
by Strahil 25 Jul '19

25 Jul '19

According to another post in the mailing list, the Engine Hosts (that has ovirt-ha-agent/ovirt-ha-broker running) is checking http://{fqdn}/ovirt-engine/services/health As the IP is changed, I think you need to check the URL before and after thr mifgration. Best Regards, Strahil NikolovOn Jul 23, 2019 16:41, Derek Atkins <derek(a)ihtfp.com> wrote: > > Hi, > > If I understand it correctly, the HE Hosts try to ping (or SSH, or > otherwise reach) the Engine host. If it reaches it, then it passes the > liveness check. If it cannot reach it, then it fails. So to me this error > means that there is some configuration, somewhere, that is trying to reach > the engine on the old address (which fails when the engine has the new > address). > > I do not know where in the *host* configuration this data lives, so I > cannot suggest where you need to change it. > > Can 10.16.248.x reach 10.8.236.x and vice-versa? > > Maybe multi-home the engine on both networks for now until you figure it out? > > -derek > > On Tue, July 23, 2019 9:13 am, carl langlois wrote: > > Hi, > > > > We have managed to stabilize the DNS udpate in out network. Now the > > current > > situation is. > > I have 3 hosts that can run the engine (hosted-engine). > > They were all in the 10.8.236.x. Now i have moved one of them in the > > 10.16.248.x. > > > > If i boot the engine on one of the host that is in the 10.8.236.x the > > engine is going up with status "good". I can access the engine UI. I can > > see all my hosts even the one in the 10.16.248.x network. > > > > But if i boot the engine on the hosted-engine host that was switch to the > > 10.16.248.x the engine is booting. I can ssh to it but the status is > > always > > " fail for liveliness check". > > The main difference is that when i boot on the host that is in the > > 10.16.248.x network the engine gets a address in the 248.x network. > > > > On the engine i have this in the > > /var/log/ovirt-engine-dwh/ovirt-engine-dwhd.log > > 019-07-23 > > 09:05:30|MFzehi|YYTDiS|jTq2w8|OVIRT_ENGINE_DWH|SampleTimeKeepingJob|Default|5|tWarn|tWarn_1|Can > > not sample data, oVirt Engine is not updating the statistics. Please check > > your oVirt Engine status.|9704 > > the engine.log seems okey. > > > > So i need to understand what this " liveliness check" do(or try to do) so > > i > > can investigate why the engine status is not becoming good. > > > > The initial deployment was done in the 10.8.236.x network. Maybe is as > > something to do with that. > > > > Thanks & Regards > > > > Carl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 18, 2019 at 8:53 AM Miguel Duarte de Mora Barroso < > > mdbarroso(a)redhat.com> wrote: > > > >> On Thu, Jul 18, 2019 at 2:50 PM Miguel Duarte de Mora Barroso > >> <mdbarroso(a)redhat.com> wrote: > >> > > >> > On Thu, Jul 18, 2019 at 1:57 PM carl langlois <crl.langlois(a)gmail.com> > >> wrote: > >> > > > >> > > Hi Miguel, > >> > > > >> > > I have managed to change the config for the ovn-controler. > >> > > with those commands > >> > > ovs-vsctl set Open_vSwitch . external-ids:ovn-remote=ssl: > >> 10.16.248.74:6642 > >> > > ovs-vsctl set Open_vSwitch . external-ids:ovn-encap-ip=10.16.248.65 > >> > > and restating the services > >> > > >> > Yes, that's what the script is supposed to do, check [0]. > >> > > >> > Not sure why running vdsm-tool didn't work for you. > >> > > >> > > > >> > > But even with this i still have the "fail for liveliness check" when > >> starting the ovirt engine. But one thing i notice with our new network > >> is > >> that the reverse DNS does not work(IP -> hostname). The forward is > >> working > >> fine. I am trying to see with our IT why it is not working. > >> > > >> > Do you guys use OVN? If not, you could disable the provider, install > >> > the hosted-engine VM, then, if needed, re-add / re-activate it . > >> > >> I'm assuming it fails for the same reason you've stated initially - > >> i.e. ovn-controller is involved; if it is not, disregard this msg :) > >> > > >> > [0] - > >> https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/setu… > >> > > >> > > > >> > > Regards. > >> > > Carl > >>

4 6

Active-Passive DR: mutual for different storage domains possible?
by Gianluca Cecchi 25 Jul '19

25 Jul '19

Hello, suppose I want to implement Active-Passive DR between 2 sites. Sites are SiteA and SiteB. I have 2 storage domains SD1 and SD2, that I can configure so that SD1 is active in storage array installed in SiteA with replica in SiteB and SD2 the reverse. I have 4 hosts: host1 and host2 in SiteA and host3 and host4 in SiteB. I would like to optimize compute resources and workload so that: oVirt env OV1 with ovmgr1 in SiteA (external engine) is composed by host1 and host2 and configured with SD1 oVirt env OV2 with ovmgr2 in SiteB (external engine) is composed by host3 and host4 and configured with SD2. Can I use OV2 as DR for OV1 for VMs installed on SD1 and at the same time OV1 as DR for OV2 for VMs installed on SD2? Thanks in advance, Gianluca Cecchi

2 5

Failed Installation
by Marcello Gentile 25 Jul '19

25 Jul '19

I’m installing a self-hosted engine on CentOS7.6 using cockpit. Keep getting this error [ INFO ] TASK [ovirt.hosted_engine_setup : Wait for the host to be up] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Check host status] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, please check engine logs, fix accordingly and re-deploy.\n"} [ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost -> localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n”} The logs don’t tell me much or at least the ones I’m checking Any help would be greatly appreciated Thanks

2 1

iSCSI-based storages won't login to the portals with all IPs on reboot
by nicolas＠devels.es 25 Jul '19

25 Jul '19

Hi, We're running oVirt 4.3.2. Currently, we have one storage backend (cabinet) with two controllers, each of them with 2 network interfaces (4 network interfaces in total). When we added the Storage Domain, we discovered the target for each of the 4 IPs and marked the LUN so it would be added with 4 different IPs. When we put a host on maintenance, all the paths are deactivated, and when we activate it back it discovers all the 4 paths for the storage backend. However, if we reboot the host, on activation it only activates one path. We can see this running 'multipath -ll'. We can manually activate the rest of the paths using this command for each of the IPs: # iscsiadm --mode discovery --type sendtargets --portal 10.X.X.X --login However, we wonder why oVirt wouldn't log into each of the IPs upon a boot. Is there something we're missing? Can this be fixed manually? Currently we're running a script on boot that will issue the command above for each of the IPs of the cabinet. Thanks for any help!

2 1

template permissions not inherited (4.3.4)
by Timmi 24 Jul '19

24 Jul '19

Hi oVirt List, I have just a quick question if I should open a ticket for this or if I'm doing something wrong. I created a new VM template with specific permissions in addition to the system wide permissions. If I create a new VM with the template I notices that only system permissions are copied to the permission of the new VM. Is this the intended behavior? I was somehow under the impression that the permission from the template should have been copied to the newly created VM. Tested with Version 4.3.4.3-1.el7 Best regards Christoph

4 6

if hosted engine corrupted
by Crazy Ayansh 24 Jul '19

24 Jul '19

Hi All, What is the way if self hosted engine corrupted in any environment and we need to setup it from scratch however on the new engine we need the same hosts and virtual machine without loosing data. Thanks Shashakn

2 1

reinstallation information
by nikkognt＠gmail.com 24 Jul '19

24 Jul '19

Hi, after a blackout one host of my ovirt not working properly. I tried to Reinstall it but ends with the following error "Failed to install Host ov1. Failed to execute stage 'Closing up': Failed to start service 'vdsmd'." I tried to start manually but it not start . Now, I would like to reinstall from iso ovirt node. After I put the host is in maintenance must I remove host from the cluster (Hosts -> host1 -> Remove) or I can reinstall without remove it? If I remore it from the cluster the network configurations I lose them or not? My ovirt version is oVirt Engine Version: 4.1.9.1-1.el7.centos.

2 6

Re: Storage domain 'Inactive' but still functional
by Strahil 24 Jul '19

24 Jul '19

I forgot to mention that LVM config has to be modified in order to 'inform' local LVM stack to rely on clvmd/dlm for locking purposes. Yet, this brings abother layer of complexity which I prefer to avoid , thus I use HA-LVM on my pacemaker clusters. @Martin, Check the link from Benny and if possible check if the 2 cases are related. Best Regards, Strahil NikolovOn Jul 24, 2019 11:07, Benny Zlotnik <bzlotnik(a)redhat.com> wrote: > > We have seen something similar in the past and patches were posted to deal with this issue, but it's still in progress[1] > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1553133 > > On Mon, Jul 22, 2019 at 8:07 PM Strahil <hunter86_bg(a)yahoo.com> wrote: >> >> I have a theory... But after all without any proof it will remain theory. >> >> The storage volumes are just VGs over a shared storage.The SPM host is supposed to be the only one that is working with the LVM metadata, but I have observed that when someone is executing a simple LVM command (for example -lvs, vgs or pvs ) while another one is going on on another host - your metadata can corrupt, due to lack of clvmd. >> >> As a protection, I could offer you to try the following solution: >> 1. Create new iSCSI lun >> 2. Share it to all nodes and create the storage domain. Set it to maintenance. >> 3. Start dlm & clvmd services on all hosts >> 4. Convert the VG of your shared storage domain to have a 'cluster'-ed flag: >> vgchange -c y mynewVG >> 5. Check the lvs of that VG. >> 6. Activate the storage domain. >> >> Of course test it on a test cluster before inplementing it on Prod. >> This is one of the approaches used in Linux HA clusters in order to avoid LVM metadata corruption. >> >> Best Regards, >> Strahil Nikolov >> >> On Jul 22, 2019 15:46, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> wrote: >>> >>> Hi, >>> >>> Op 22-7-2019 om 14:30 schreef Strahil: >>>> >>>> If you can give directions (some kind of history) , the dev might try to reproduce this type of issue. >>>> >>>> If it is reproduceable - a fix can be provided. >>>> >>>> Based on my experience, if something as used as Linux LVM gets broken, the case is way hard to reproduce. >>> >>> >>> Yes, I'd think so too, especially since this activity (online moving of disk images) is done all the time, mostly without problems. In this case, there was a lot of activity on all storage domains, because I'm moving all my storage (> 10TB in 185 disk images) to a new storage platform. During the online move of one the images, the metadata checksum became corrupted and the storage domain went offline. >>> >>> Of course, I could dig up the engine logs and vdsm logs of when it happened, but that would be some work and I'm not very confident that the actual cause would be in there. >>> >>> If any oVirt devs are interested in the logs, I'll provide them, but otherwise I think I'll just see it as an incident and move on. >>> >>> Best regards, >>> Martijn. >>> >>> >>> >>> >>> On Jul 22, 2019 10:17, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Thanks for the tips! I didn't know about 'pvmove', thanks. >>>>> >>>>> In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. >>>>> >>>>> Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) >>>>> >>>>> Thanks, >>>>> Martijn. >>>>> >>>>> >>>>> >>>>> Op 19-7-2019 om 19:06 schreef Strahil Nikolov: >>>>>> >>>>>> Hi Martin, >>>>>> >>>>>> First check what went wrong with the VG -as it could be something simple. >>>>>> vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. >>>>>> >>>>>> If you have Linux boxes - you can add disks from another storage an

1 0

Re: Storage domain 'Inactive' but still functional
by Martijn Grendelman 24 Jul '19

24 Jul '19

Op 24-7-2019 om 10:07 schreef Benny Zlotnik: We have seen something similar in the past and patches were posted to deal with this issue, but it's still in progress[1] [1] https://bugzilla.redhat.com/show_bug.cgi?id=1553133 That's some interesting reading, and it sure looks like the problem I had. Thanks! Best regards, Martijn. On Mon, Jul 22, 2019 at 8:07 PM Strahil <hunter86_bg(a)yahoo.com<mailto:hunter86_bg@yahoo.com>> wrote: I have a theory... But after all without any proof it will remain theory. The storage volumes are just VGs over a shared storage.The SPM host is supposed to be the only one that is working with the LVM metadata, but I have observed that when someone is executing a simple LVM command (for example -lvs, vgs or pvs ) while another one is going on on another host - your metadata can corrupt, due to lack of clvmd. As a protection, I could offer you to try the following solution: 1. Create new iSCSI lun 2. Share it to all nodes and create the storage domain. Set it to maintenance. 3. Start dlm & clvmd services on all hosts 4. Convert the VG of your shared storage domain to have a 'cluster'-ed flag: vgchange -c y mynewVG 5. Check the lvs of that VG. 6. Activate the storage domain. Of course test it on a test cluster before inplementing it on Prod. This is one of the approaches used in Linux HA clusters in order to avoid LVM metadata corruption. Best Regards, Strahil Nikolov On Jul 22, 2019 15:46, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl<mailto:Martijn.Grendelman@isaac.nl>> wrote: Hi, Op 22-7-2019 om 14:30 schreef Strahil: If you can give directions (some kind of history) , the dev might try to reproduce this type of issue. If it is reproduceable - a fix can be provided. Based on my experience, if something as used as Linux LVM gets broken, the case is way hard to reproduce. Yes, I'd think so too, especially since this activity (online moving of disk images) is done all the time, mostly without problems. In this case, there was a lot of activity on all storage domains, because I'm moving all my storage (> 10TB in 185 disk images) to a new storage platform. During the online move of one the images, the metadata checksum became corrupted and the storage domain went offline. Of course, I could dig up the engine logs and vdsm logs of when it happened, but that would be some work and I'm not very confident that the actual cause would be in there. If any oVirt devs are interested in the logs, I'll provide them, but otherwise I think I'll just see it as an incident and move on. Best regards, Martijn. On Jul 22, 2019 10:17, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl><mailto:Martijn.Grendelman@isaac.nl> wrote: Hi, Thanks for the tips! I didn't know about 'pvmove', thanks. In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) Thanks, Martijn. Op 19-7-2019 om 19:06 schreef Strahil Nikolov: Hi Martin, First check what went wrong with the VG -as it could be something simple. vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. If you have Linux boxes - you can add disks from another storage and then pvmove the data inside the VM. Of course , you will need to reinstall grub on the new OS disk , or you won't be able to boot afterwards. If possible, try with a test VM before proceeding with important ones. Backing up the VMs is very important , because working on LVM metada _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37UDAWDXON3UR… -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend.

1 0

Re: Storage domain 'Inactive' but still functional
by Strahil 24 Jul '19

24 Jul '19

I have a theory... But after all without any proof it will remain theory. The storage volumes are just VGs over a shared storage.The SPM host is supposed to be the only one that is working with the LVM metadata, but I have observed that when someone is executing a simple LVM command (for example -lvs, vgs or pvs ) while another one is going on on another host - your metadata can corrupt, due to lack of clvmd. As a protection, I could offer you to try the following solution: 1. Create new iSCSI lun 2. Share it to all nodes and create the storage domain. Set it to maintenance. 3. Start dlm & clvmd services on all hosts 4. Convert the VG of your shared storage domain to have a 'cluster'-ed flag: vgchange -c y mynewVG 5. Check the lvs of that VG. 6. Activate the storage domain. Of course test it on a test cluster before inplementing it on Prod. This is one of the approaches used in Linux HA clusters in order to avoid LVM metadata corruption. Best Regards, Strahil NikolovOn Jul 22, 2019 15:46, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> wrote: > > Hi, > > Op 22-7-2019 om 14:30 schreef Strahil: >> >> If you can give directions (some kind of history) , the dev might try to reproduce this type of issue. >> >> If it is reproduceable - a fix can be provided. >> >> Based on my experience, if something as used as Linux LVM gets broken, the case is way hard to reproduce. > > > Yes, I'd think so too, especially since this activity (online moving of disk images) is done all the time, mostly without problems. In this case, there was a lot of activity on all storage domains, because I'm moving all my storage (> 10TB in 185 disk images) to a new storage platform. During the online move of one the images, the metadata checksum became corrupted and the storage domain went offline. > > Of course, I could dig up the engine logs and vdsm logs of when it happened, but that would be some work and I'm not very confident that the actual cause would be in there. > > If any oVirt devs are interested in the logs, I'll provide them, but otherwise I think I'll just see it as an incident and move on. > > Best regards, > Martijn. > > > > > On Jul 22, 2019 10:17, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> wrote: >>> >>> Hi, >>> >>> Thanks for the tips! I didn't know about 'pvmove', thanks. >>> >>> In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. >>> >>> Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) >>> >>> Thanks, >>> Martijn. >>> >>> >>> >>> Op 19-7-2019 om 19:06 schreef Strahil Nikolov: >>>> >>>> Hi Martin, >>>> >>>> First check what went wrong with the VG -as it could be something simple. >>>> vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. >>>> >>>> If you have Linux boxes - you can add disks from another storage and then pvmove the data inside the VM. Of course , you will need to reinstall grub on the new OS disk , or you won't be able to boot afterwards. >>>> If possible, try with a test VM before proceeding with important ones. >>>> >>>> Backing up the VMs is very important , because working on LVM metada

2 1

USB turns off and wont come alive
by Darin Schmidt 24 Jul '19

24 Jul '19

I have 2 usb controllers installed via a riser card plugged into a M.2 slot. (Like those used for mining with GPU's). Everything works great for a long time, then my Windows 10 VM freezes and I lose all USB activity. LSPCI still sees the devices: 0a:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller 0b:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller 43:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller 44:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller I assume a and b are the same physical card as its supposed to have 2 channels each, same with 44 and 43. I tried for both 44 and 43 echo "1" > /sys/bus/pci/devices/0000\:43\:00.0/remove echo "1" > /sys/bus/pci/rescan echo "1" > /sys/bus/pci/devices/0000\:43\:00.0/reset This did not work. I cannot determine if its just the VM thats no longer seeing/using the hardware or if its the hardware itself. I wonder if its a power state thing as well? Nothing I plug into the card seems to be recognized. Any Suggestions? rebooting the VM doesnt help either. It appears the hardware is functioning, but anything you plug into it isnt being detected.

2 1

Import VM from ovirt-exported ova
by david.paning＠gmx.ch 24 Jul '19

24 Jul '19

Hello, i installed a new oVirt 4.3.4 from scratch and am attempting to restore the VMs which ran perfectly fine on my last oVirt 4.2.8, where i exported the VM to the export domain and also as an OVA file. Importing the export domain fails, which i plan to debug later. But importing the OVA file via the Administration Portal also fails with Error ID: 1153 Message: Failed to import the VM to Data Center, Cluster. The VM name is still recognised by the import routine, which tells me the file is readable... Now my questions: Is there any option to try the same from CLI, hoping to get better clues for what is going on? Are there specific log options which could give some clues? Would it be better in general to import the VM from an export-storage domain instead of using a OVA export? Any hint would highly appreciated. Best

3 2

Re: major network changes
by Strahil 23 Jul '19

23 Jul '19

Hi Carl, I think there is another thread here related to the migration to another network. As far as I know, the check liveliness try's to access the ovirt's health page. Does the new engine's ip has A/PTR record setup? Also, check the engine logs, once the HostedEngine VM is up and running. Best Regards, Strahil NikolovOn Jul 23, 2019 16:13, carl langlois <crl.langlois(a)gmail.com> wrote: > > Hi, > > We have managed to stabilize the DNS udpate in out network. Now the current situation is. > I have 3 hosts that can run the engine (hosted-engine). > They were all in the 10.8.236.x. Now i have moved one of them in the 10.16.248.x. > > If i boot the engine on one of the host that is in the 10.8.236.x the engine is going up with status "good". I can access the engine UI. I can see all my hosts even the one in the 10.16.248.x network. > > But if i boot the engine on the hosted-engine host that was switch to the 10.16.248.x the engine is booting. I can ssh to it but the status is always " fail for liveliness check". > The main difference is that when i boot on the host that is in the 10.16.248.x network the engine gets a address in the 248.x network. > > On the engine i have this in the /var/log/ovirt-engine-dwh/ovirt-engine-dwhd.log > 019-07-23 09:05:30|MFzehi|YYTDiS|jTq2w8|OVIRT_ENGINE_DWH|SampleTimeKeepingJob|Default|5|tWarn|tWarn_1|Can not sample data, oVirt Engine is not updating the statistics. Please check your oVirt Engine status.|9704 > the engine.log seems okey. > > So i need to understand what this " liveliness check" do(or try to do) so i can investigate why the engine status is not becoming good. > > The initial deployment was done in the 10.8.236.x network. Maybe is as something to do with that. > > Thanks & Regards > > Carl > > > > > > > > > > > > > > > > > > > On Thu, Jul 18, 2019 at 8:53 AM Miguel Duarte de Mora Barroso <mdbarroso(a)redhat.com> wrote: >> >> On Thu, Jul 18, 2019 at 2:50 PM Miguel Duarte de Mora Barroso >> <mdbarroso(a)redhat.com> wrote: >> > >> > On Thu, Jul 18, 2019 at 1:57 PM carl langlois <crl.langlois(a)gmail.com> wrote: >> > > >> > > Hi Miguel, >> > > >> > > I have managed to change the config for the ovn-controler. >> > > with those commands >> > > ovs-vsctl set Open_vSwitch . external-ids:ovn-remote=ssl:10.16.248.74:6642 >> > > ovs-vsctl set Open_vSwitch . external-ids:ovn-encap-ip=10.16.248.65 >> > > and restating the services >> > >> > Yes, that's what the script is supposed to do, check [0]. >> > >> > Not sure why running vdsm-tool didn't work for you. >> > >> > > >> > > But even with this i still have the "fail for liveliness check" when starting the ovirt engine. But one thing i notice with our new network is that the reverse DNS does not work(IP -> hostname). The forward is working fine. I am trying to see with our IT why it is not working. >> > >> > Do you guys use OVN? If not, you could disable the provider, install >> > the hosted-engine VM, then, if needed, re-add / re-activate it . >> >> I'm assuming it fails for the same reason you've stated initially - >> i.e. ovn-controller is involved; if it is not, disregard this msg :) >> > >> > [0] - https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/setu… >> > >> > > >> > > Regards. >> > > Carl >> > > >> > > On Thu, Jul 18, 2019 at 4:03 AM Miguel Duarte de Mora Barroso <mdbarroso(a)redhat.com> wrote: >> > >> >> > >> On Wed, Jul 17, 2019 at 7:07 PM carl langlois <crl.langlois(a)gmail.com> wrote: >> > >> > >> > >> > Hi >> > >> > Here is the output of the command >> > >> > >> > >> > [root@ovhost1 ~]# vdsm-tool --vvverbose ovn-config 10.16.248.74 ovirtmgmt >> > >> > MainThread::DEBUG::2019-07-17 13:02:52,

1 0

Stuck in "Finalizing" disk upload phase
by Vrgotic, Marko 23 Jul '19

23 Jul '19

Dear oVIrt, I initiated upload of qcow2 disk image for Centos 6.5: It reached finalizing phase and than started throwing following errors: 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. Help! oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic ActiveVideo

3 17

major network changes
by carl langlois 23 Jul '19

23 Jul '19

Hi We are in a process of changing our network connection. Our current network is using 10.8.256.x and we will change to 10.16.248.x. We have a HA ovirt cluster (around 10 nodes) currently configure on the 10.8.256.x. So my question is is it possible to relocate the ovirt cluster to the 10.16.248.x. We have tried to move everything to the new network without success. All the node seem to boot up properly, our gluster storage also work properly. When we try to start the hosted-engine it goes up but fail the liveliness check. We have notice in the /var/log/openvswitch/ovn-controller.log that he is triying to connect to the hold ip address of the hosted-engine vm. 019-07-16T18:41:29.483Z|01992|reconnect|INFO|ssl:10.8.236.244:6642: waiting 8 seconds before reconnect 2019-07-16T18:41:37.489Z|01993|reconnect|INFO|ssl:10.8.236.244:6642: connecting... 2019-07-16T18:41:45.497Z|01994|reconnect|INFO|ssl:10.8.236.244:6642: connection attempt timed out So my question is were is the 10.8.236.244 come from. The routing table for one of our host look like this estination Gateway Genmask Flags Metric Ref Use Iface default gateway 0.0.0.0 UG 0 0 0 ovirtmgmt 10.16.248.0 0.0.0.0 255.255.255.0 U 0 0 0 ovirtmgmt link-local 0.0.0.0 255.255.0.0 U 1002 0 0 eno1 link-local 0.0.0.0 255.255.0.0 U 1003 0 0 eno2 link-local 0.0.0.0 255.255.0.0 U 1025 0 0 ovirtmgmt Any help would be really appreciated. Regards Carl

3 12

Attach the snapshot to the backup virtual machine and activate the disk
by smidhunraj＠gmail.com 23 Jul '19

23 Jul '19

This is the doubt regarding to the rest api <https://ovirt.org/documentation/admin-guide/chap-Backups_and_Migration.html…> Can you please tell me what would be the response of the api request of step 4 ((will it be a success or a error)) ===================================================================== Attach the snapshot to the backup virtual machine and activate the disk: POST /api/vms/22222222-2222-2222-2222-222222222222/disks/ HTTP/1.1 Accept: application/xml Content-type: application/xml <disk id="11111111-1111-1111-1111-111111111111"> <snapshot id="11111111-1111-1111-1111-111111111111"/> <active>true</active> </disk> ============================================================================================== if we are setting up a bare VM without any backup mechanism in it and try to attach a vm to it.

1 0

Re: ovirt-engine-appliance ova
by Strahil 23 Jul '19

23 Jul '19

It looks like a kickstart error. Maybe a parameter in the kickstart is missing ? Best Regards, Strahil NikolovOn Jul 23, 2019 09:08, Yedidyah Bar David <didi(a)redhat.com> wrote: > > On Mon, Jul 22, 2019 at 11:53 PM Jingjie Jiang <jingjie.jiang(a)oracle.com> wrote: >> >> Hi David, > > > (Actually it's "Yedidyah" or "Didi") > >> >> Thanks for your info. >> >> Please check my reply inline. >> >> >> -Jingjie >> >> On 7/16/19 3:55 AM, Yedidyah Bar David wrote: >>> >>> On Thu, Jul 11, 2019 at 10:46 PM <jingjie.jiang(a)oracle.com> wrote: >>> >>>> Hi, >>>> >>>> Can someone tell me how to generate ovirt-engine-appliance ova file in ovirt-engine-appliance-4.3-20190610.1.el7.x86_64.rpm? >>>> >>> You might want to check the project's source code: >>> >>> >>> >>> https://github.com/ovirt/ovirt-appliance/ >>> >>> >>> >>> Or study the logs of a CI build of it: >>> >>> >>> >>> https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el7-x8… >>> >>> >>> >>> I never tried building it myself locally, though. >> >> I tried to build after checked out the source code from https://github.com/ovirt/ovirt-appliance/, >> >> but the build failed. >> >> # make >> livemedia-creator --make-disk --ram=2048 --vcpus=4 --iso=boot.iso --ks=ovirt-engine-appliance.ks --qcow2 --image-name=ovirt-engine-appliance.qcow2 >> 2019-07-22 12:34:00,095: livemedia-creator 19.7.19-1 >> 2019-07-22 12:34:00,154: disk_size = 51GiB >> 2019-07-22 12:34:00,154: disk_img = /var/tmp/ovirt-engine-appliance.qcow2 >> 2019-07-22 12:34:00,154: install_log = /root/ovirt/ovirt-appliance/engine-appliance/virt-install.log >> mount: /dev/loop0 is write-protected, mounting read-only >> Formatting '/var/tmp/ovirt-engine-appliance.qcow2', fmt=qcow2 size=54760833024 encryption=off cluster_size=65536 lazy_refcounts=off >> 2019-07-22 12:34:10,195: Running virt-install. >> >> Starting install... >> Retrieving file vmlinuz... | 6.3 MB 00:00 >> Retrieving file initrd.img... | 50 MB 00:00 >> Domain installation still in progress. You can reconnect to >> the console to complete the installation process. >> ...... >> 2019-07-22 12:35:15,281: Installation error detected. See logfile. >> 2019-07-22 12:35:15,283: Shutting down LiveOS-27f2dc2b-4b30-4eb1-adcd-b5ab50fdbf55 >> Domain LiveOS-27f2dc2b-4b30-4eb1-adcd-b5ab50fdbf55 destroyed >> >> Domain LiveOS-27f2dc2b-4b30-4eb1-adcd-b5ab50fdbf55 has been undefined >> >> 2019-07-22 12:35:15,599: unmounting the iso >> 2019-07-22 12:35:20,612: Install failed: virt_install failed >> 2019-07-22 12:35:20,613: Removing bad disk image >> 2019-07-22 12:35:20,613: virt_install failed >> make: *** [ovirt-engine-appliance.qcow2] Error 1 >> >> In the log I found the error as following from virt-install.log: >> >> 16:35:07,472 ERR anaconda:CmdlineError: The following mandatory spokes are not completed:#012Installation source#012Software selection >> 16:35:07,472 DEBUG anaconda:running handleException >> 16:35:07,473 CRIT anaconda:Traceback (most recent call last):#012#012 File "/usr/lib64/python2.7/site-packages/pyanaconda/ui/tui/simpleline/base.py", line 352, in _mainloop#012 prompt = last_screen.prompt(self._screens[-1][1])#012#012 File "/usr/lib64/python2.7/site-packages/pyanaconda/ui/tui/hubs/summary.py", line 107, in prompt#012 raise CmdlineError(errtxt)#012#012CmdlineError: The following mandatory spokes are not completed:#012Installation source#012Software selection >> 16:35:08,020 DEBUG anaconda:Gtk cannot be initialized >> 16:35:08,020 DEBUG anaconda:In the main thread, running exception handler >> 16:35:08,386 NOTICE multipathd:zram0: add path (uevent) >> 16:35:08,386 NOTICE multipathd:zram0: spurious uevent, path already in pathvec >> 16:35:08,386 NOTICE multipathd:zram0: HDIO_GETGEO failed with 25 >> 16:35:08,386 ERR multipathd:zram0: failed to get path uid >> 16:35:08,388 ERR multipathd:uevent trigger error >> >> Can you help me to fix the issue? > > > Sorry, I never tried to build it myself, nor have experience with livemedia-creator. As I wrote above, I suggest to compare your output/result with that of oVirt CI. Otherwise, I'd probably start debugging by searching the net for the error messages you received. > > Good luck and best regards, > >> >> >>>> I tried to import ovirt-engine-appliance ova(ovirt-engine-appliance-4.3-20190610.1.el7.ova) from ovirt-engine, but I got error as following: >>>> >>>> Failed to load VM configuration from OVA file: /var/tmp/ovirt-engine-appliance-4.2-20190121.1.el7.ova >>>> >>> No idea why this failed. >>> >>> >>> >>>> I guess ovirt-engine-appliance-4.2-20190121.1.el7.ova has more than CentOS7.6. >>>> >>> It has CentOS + oVirt engine. >>> >>> >>> >>> The only major use for it is by hosted-engine --deploy. In theory you >>> >>> can try importing it elsewhere, but I do not recall reports about >>> >>> people that tried this and whether it works. >>> >>> >>> >>> Best regards, >>> > > > -- > Didi

1 0

guest agent w10
by suporte＠logicworks.pt 22 Jul '19

22 Jul '19

Hello, I installed Qemu guest agent 7.4.5 on windows 10. The service is running, but I cannot see the IP address or the FQDN on the engine. I'm running oVirt 4.3.4.3-1.el7 Am I forgetting something? Thanks -- Jose Ferradeira http://www.logicworks.pt

2 2

Request for clarification of logline
by Vrgotic, Marko 22 Jul '19

22 Jul '19

Dear oVirt, I was not able to find out meaning behind following WARN message: WARN [org.ovirt.engine.core.vdsbroker.libvirt.VmDevicesConverter] (EE-ManagedThreadFactory-engineScheduled-Thread-50) [] unmanaged disk with path '/var/run/vdsm/payload/0687ad1a-4827-48d9-8629-b29a000df280.760410382a41ff5e6b1ee9365856517e.img' is ignored Will somebody be so kind to clarify the actual meaning of it or point me in right direction of cause? I can provide more information if required. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic ActiveVideo

3 4

Cinderlib managed block storage, ceph jewel
by mathias.schwenke＠uni-dortmund.de 22 Jul '19

22 Jul '19

Hi. I tried to use manged block storage to connect our oVirt cluster (Version 4.3.4.3-1.el7) to our ceph storage ( version 10.2.11). I used the instructions from https://ovirt.org/develop/release-management/features/storage/cinderlib-int… At the moment, in the ovirt administration portal I can create and delete ceph volumes (ovirt disks) and attach them to virtual machines. If I try to launch a vm with connected ceph block storage volume, starting fails: 2019-07-16 19:39:09,251+02 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-53) [7cada945] Unexpected return value: Status [code=926, message=Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/libexec/vdsm/managedvolume-helper\', \'attach\'] failed with rc=1 out=\'\' err=\'oslo.privsep.daemon: Running privsep helper: [\\\'sudo\\\', \\\'privsep-helper\\\', \\\'--privsep_context\\\', \\\'os_brick.privileged.default\\\', \\\'--privsep_sock_path\\\', \\\'/tmp/tmpB6ZBAs/privsep.sock\\\']\\noslo.privsep.daemon: Spawned new privsep daemon via rootwrap\\noslo.privsep.daemon: privsep daemon starting\\noslo.privsep.daemon: privsep process running with uid/gid: 0/0\\noslo.privsep.daemon: privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none\\noslo.privsep.daemon: privsep daemon running as pid 112531\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-help er", line 154, in <module>\\n sys.exit(main(sys.argv[1:]))\\n File "/usr/libexec/vdsm/managedvolume-helper", line 77, in main\\n args.command(args)\\n File "/usr/libexec/vdsm/managedvolume-helper", line 137, in attach\\n attachment = conn.connect_volume(conn_info[\\\'data\\\'])\\n File "/usr/lib/python2.7/site-packages/vdsm/storage/nos_brick.py", line 96, in connect_volume\\n run_as_root=True)\\n File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute\\n result = self.__execute(*args, **kwargs)\\n File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute\\n return execute_root(*cmd, **kwargs)\\n File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 205, in _wrap\\n return self.channel.remote_call(name, args, kwargs)\\n File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call\\n raise exc_type(*result[2])\\noslo_concurrency.processutils.Pr ocessExecutionError: Unexpected error while running command.\\nCommand: rbd map volume-a57dbd5c-2f66-460f-b37f-5f7dfa95d254 --pool ovirt-volumes --conf /tmp/brickrbd_TLMTkR --id ovirtcinderlib --mon_host 192.168.61.1:6789 --mon_host 192.168.61.2:6789 --mon_host 192.168.61.3:6789\\nExit code: 6\\nStdout: u\\\'RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".\\\\nIn some cases useful info is found in syslog - try "dmesg | tail" or so.\\\\n\\\'\\nStderr: u\\\'rbd: sysfs write failed\\\\nrbd: map failed: (6) No such device or address\\\\n\\\'\\n\'',)] 2019-07-16 19:39:09,251+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.AttachManagedBlockStorageVolumeVDSCommand] (default task-53) [7cada945] Failed in 'AttachManagedBlockStorageVolumeVDS' method After disconnecting the disk, I can delete it (the volume disappears from ceph), but the disks stays in my oVirt administration portal as cinderlib means the disk ist still connected: 2019-07-16 19:42:53,551+02 INFO [org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand] (EE-ManagedThreadFactory-engine-Thread-487362) [887b4d11-302f-4f8d-a3f9-7443a80a47ba] Running command: RemoveDiskCommand internal: false. Entities affected : ID: a57dbd5c-2f66-460f-b37f-5f7dfa95d254 Type: DiskAction group DELETE_DISK with role type USER 2019-07-16 19:42:53,559+02 INFO [org.ovirt.engine.core.bll.storage.disk.managedblock.RemoveManagedBlockStorageDiskCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [] Running command: RemoveManagedBlockStorageDiskCommand internal: true. 2019-07-16 19:42:56,240+02 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [] cinderlib execution failed DBReferenceError: (psycopg2.IntegrityError) update or delete on table "volumes" violates foreign key constraint "volume_attachment_volume_id_fkey" on table "volume_attachment" 2019-07-16 19:42:55,958 - cinderlib-client - INFO - Deleting volume 'a57dbd5c-2f66-460f-b37f-5f7dfa95d254' [887b4d11-302f-4f8d-a3f9-7443a80a47ba] 2019-07-16 19:42:56,099 - cinderlib-client - ERROR - Failure occurred when trying to run command 'delete_volume': (psycopg2.IntegrityError) update or delete on table "volumes" violates foreign key constraint "volume_attachment_volume_id_fkey" on table "volume_attachment" DETAIL: Key (id)=(a57dbd5c-2f66-460f-b37f-5f7dfa95d254) is still referenced from table "volume_attachment". [SQL: 'DELETE FROM volumes WHERE volumes.deleted = false AND volumes.id = %(id_1)s'] [parameters: {'id_1': u'a57dbd5c-2f66-460f-b37f-5f7dfa95d254'}] [887b4d11-302f-4f8d-a3f9-7443a80a47ba] How are your experiences with oVirt, cinderlib and ceph? Should it work?

3 3

Change vDisk ID
by Николаев Алексей 22 Jul '19

22 Jul '19

1 0

Re: Storage domain 'Inactive' but still functional
by Strahil 22 Jul '19

22 Jul '19

If you can give directions (some kind of history) , the dev might try to reproduce this type of issue. If it is reproduceable - a fix can be provided. Based on my experience, if something as used as Linux LVM gets broken, the case is way hard to reproduce. Best Regards, Strahil NikolovOn Jul 22, 2019 10:17, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> wrote: > > Hi, > > Thanks for the tips! I didn't know about 'pvmove', thanks. > > In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. > > Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) > > Thanks, > Martijn. > > > > Op 19-7-2019 om 19:06 schreef Strahil Nikolov: >> >> Hi Martin, >> >> First check what went wrong with the VG -as it could be something simple. >> vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. >> >> If you have Linux boxes - you can add disks from another storage and then pvmove the data inside the VM. Of course , you will need to reinstall grub on the new OS disk , or you won't be able to boot afterwards. >> If possible, try with a test VM before proceeding with important ones. >> >> Backing up the VMs is very important , because working on LVM metadata is quite risky. >> Last time I had such an issue , I was working on clustered LVs which got their PVs "Missing". For me , restore from VG backup fixed the issue - but that might not be always the case. >> >> Just get the vgcfgbackup's output and compare with diff or vimdiff and check what is different. >> >> Sadly, I think that this is more a Linux problem , than an oVirt problem. >> >> Best Regards, >> Strahil Nikolov >> >> В четвъртък, 18 юли 2019 г., 18:51:32 ч. Гринуич+3, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> написа: >> >> >> Hi! >> >> Thanks. Like I wrote, I have metadata backups from /etc/lvm/backup and -/archive, and I also have the current metadata as it exis

1 0

LiveStoreageMigration failed
by Christoph Köhler 22 Jul '19

22 Jul '19

Hello, I try to migrate a disk of a running vm from gluster 3.12.15 to gluster 3.12.15 but it fails. libGfApi set to true by engine-config. ° taking a snapshot first, is working. Then at engine-log: 2019-07-18 09:29:13,932+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Failed in 'VmReplicateDiskStartVDS' method 2019-07-18 09:29:13,936+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovvirt07 command VmReplicateDiskStartVDS failed: Drive replication error 2019-07-18 09:29:13,936+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand' return value 'StatusOnlyReturn [status=Status [code=55, message=Drive replication error]]' 2019-07-18 09:29:13,936+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] HostName = ovvirt07 2019-07-18 09:29:13,937+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'VmReplicateDiskStartVDSCommand(HostName = ovvirt07, VmReplicateDiskParameters:{hostId='3a7bf85c-e92d-4559-908e-5eed2f5608d4', vmId='3b79d0c0-47e9-47c3-8511-980a8cfe147c', storagePoolId='00000001-0001-0001-0001-000000000311', srcStorageDomainId='e54d835a-d8a5-44ae-8e17-fcba1c54e46f', targetStorageDomainId='4dabb6d6-4be5-458c-811d-6d5e87699640', imageGroupId='d2964ff9-10f7-4b92-8327-d68f3cfd5b50', imageId='62656632-8984-4b7e-8be1-fd2547ca0f98'})' execution failed: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Drive replication error, code = 55 2019-07-18 09:29:13,937+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] FINISH, VmReplicateDiskStartVDSCommand, return: , log id: 5b2afb0b 2019-07-18 09:29:13,937+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Failed VmReplicateDiskStart (Disk 'd2964ff9-10f7-4b92-8327-d68f3cfd5b50' , VM '3b79d0c0-47e9-47c3-8511-980a8cfe147c') 2019-07-18 09:29:13,938+02 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'LiveMigrateDisk' id: '8174c74c-8ab0-49fa-abfc-44d8b7c691e0' with children [03672b60-443b-47ba-834c-ac306d7129d0, 562522fc-6691-47fe-93bf-ef2c45e85676] failed when attempting to perform the next operation, marking as 'ACTIVE' 2019-07-18 09:29:13,938+02 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] EngineException: Drive replication error (Failed with error replicaErr and code 55): org.ovirt.engine.core.common.errors.EngineException: EngineException: Drive replication error (Failed with error replicaErr and code 55) at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.replicateDiskStart(LiveMigrateDiskCommand.java:526) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.performNextOperation(LiveMigrateDiskCommand.java:233) [bll.jar:] at org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:77) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_212] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2019-07-18 09:29:13,938+02 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'LiveMigrateDisk' id: '8174c74c-8ab0-49fa-abfc-44d8b7c691e0' child commands '[03672b60-443b-47ba-834c-ac306d7129d0, 562522fc-6691-47fe-93bf-ef2c45e85676]' executions were completed, status 'FAILED' 2019-07-18 09:29:15,019+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand' with failure. 2019-07-18 09:29:15,019+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Failed during live storage migration of disk 'd2964ff9-10f7-4b92-8327-d68f3cfd5b50' of vm '3b79d0c0-47e9-47c3-8511-980a8cfe147c', attempting to end replication before deleting the target disk // LiveStorageMigration on gluster - should that work at all? Has someone tried it? Greetings! Christoph Köhler

4 5

Re: Storage domain 'Inactive' but still functional
by Martijn Grendelman 22 Jul '19

22 Jul '19

Hi, Thanks for the tips! I didn't know about 'pvmove', thanks. In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) Thanks, Martijn. Op 19-7-2019 om 19:06 schreef Strahil Nikolov: Hi Martin, First check what went wrong with the VG -as it could be something simple. vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. If you have Linux boxes - you can add disks from another storage and then pvmove the data inside the VM. Of course , you will need to reinstall grub on the new OS disk , or you won't be able to boot afterwards. If possible, try with a test VM before proceeding with important ones. Backing up the VMs is very important , because working on LVM metadata is quite risky. Last time I had such an issue , I was working on clustered LVs which got their PVs "Missing". For me , restore from VG backup fixed the issue - but that might not be always the case. Just get the vgcfgbackup's output and compare with diff or vimdiff and check what is different. Sadly, I think that this is more a Linux problem , than an oVirt problem. Best Regards, Strahil Nikolov В четвъртък, 18 юли 2019 г., 18:51:32 ч. Гринуич+3, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl><mailto:Martijn.Grendelman@isaac.nl> написа: Hi! Thanks. Like I wrote, I have metadata backups from /etc/lvm/backup and -/archive, and I also have the current metadata as it exists on disk. What I'm most concerned about, is the proposed procedure. I would create a backup of the VG, but I'm not sure what would be the most sensible way to do it. I could make a new iSCSI target and simply 'dd' the whole disk over, but that would take quite some time (it's 2,5 TB) and there are VMs that can't really be down for that long. And I'm not even sure that dd'ing the disk like that is a sensible strategy. Moving disks out of the domain is currently not possible. oVirt says 'Source Storage Domain is not active'. Thanks, Martijn. Op 18-7-2019 om 17:44 schreef Strahil Nikolov: Can you check the /etc/lvm/backup and /etc/lvm/archive on your SPM host (check the other hosts, just in case you find anything useful) ? Usually LVM makes backup of everything. I would recommend you to: 1. Create a backup of the problematic VG 2. Compare the backup file and a file from backup/archive folders for the same VG Check what is different with diff/vimdiff . It might give you a clue. I had some issues (non-related to oVirt) and restoring the VG from older backup did help me .Still ,any operation on block devices should be considered risky and a proper backup is needed. You could try to move a less important VM's disks out of this storage domain to another one. If it succeeds - then you can evacuate all VMs away before you can start "breaking" the storage domain. Best Regards, Strahil Nikolov В четвъртък, 18 юли 2019 г., 16:59:46 ч. Гринуич+3, Martijn Grendelman <martijn.grendelman(a)isaac.nl><mailto:martijn.grendelman@isaac.nl> написа: Hi, It appears that O365 has trouble delivering mails to this list, so two earlier mails of mine are still somewhere in a queue and may yet be delivered. This mail has all of the content of 3 successive mails. I apologize for this format. Op 18-7-2019 om 11:20 schreef Martijn Grendelman: Op 18-7-2019 om 10:16 schreef Martijn Grendelman: Hi, For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I need some help. Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a disk image off of it, to another domain. This failed, and somewhere in the process, the whole storage domain went to status 'Inactive'. From engine.log: 2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'. 2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2019-07-17 16:30:35,410+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) was deactivated by system because it's not visible by any of the hosts. The thing is, the domain is still functional on all my hosts. It carries over 50 disks, and all involved VMs are up and running, and don't seem to have any problems. Also, 'iscsiadm' on all hosts seems to indiciate that everything is fine with this specific target and reading from the device with dd, or getting its size with 'blockdev' all works without issue. When I try to reactivate the domain, these errors are logged: 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',), code = 358 2019-07-18 09:34:53,648+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) by martijn@-authz On the SPM host, there are errors that indicate problems with the LVM volume group: 2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429', spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48) 2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool] sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78 (sp:1125) 2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=[' /dev/mapper/23536316636393463: Checksum error at offset 2748693688832', " Couldn't read volume group metadata from /dev/mapper/23536316636393463.", ' Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/23536316636393463', ' Failed to scan VG from /dev/mapper/23536316636393463', ' Volume group "875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted: "Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83) I need help getting this storage domain back online. Can anyone here help me? If you need any additional information, please let me know! It appears the VG metadata is corrupt: /dev/mapper/23536316636393463: Checksum error at offset 2748693688832 Couldn't read volume group metadata from /dev/mapper/23536316636393463. Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/23536316636393463 Failed to scan VG from /dev/mapper/23536316636393463 Is this fixable? If so, how? So, I have found some information online, that suggests that PV metadata can be fixed by recreating the PV label using the correct PVID and a backup of the LVM metadata, like so: pvcreate -u <pv_uuid> --restorefile <lvm_metadata_backup> /dev/mapper/23536316636393463 Now I have the following two files: * An LVM metadata backup of yesterday 10:35, about 6 hours before the problem occurred. * The actual metadata found on the PV at offset 2748693688832 (obtained with hexedit on the block device). These are largely the same, but there are differences: * seqno = 1854 in the backup and 1865 in the actual metadata. * 3 logical volumes that are not present in the backup, but are in the actual metadata. I suspect that these are related to snapshots that were created for live storage migration, but I am not sure. In any case, I did NOT create any new disk images on this domain, so that can't be it. Now, support I wanted to try the 'pvcreate' route, then: * what would be the chances of success? Is this procedure at all advisable, or is there an alternative? * which restore file (1854 or 1865) should I use for the restore? * can I do this while the VG is in use? I tried running the command without --force, and it said 'Can't open /dev/mapper/23536316636393463 exclusively. Mounted filesystem?'. I didn't dare to try it with '--force'. I could really use some advice on how to proceed. This are about 36 VMs that have one or more disks on this domain. I could bring them down, although doing so for extended amounts of time would be problematic. I want to be careful, obviously, especially since the actual storage doesn't seem to be impacted at this time. The VMs are all still running without issue, and if I'm about to embark on a dangerous journey that could cause data loss, I need a contingency / recovery plan. Hoping someone can help... Best regards, Martijn Grendelman _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKWOTMY6MKDEQ… -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend. -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend.

1 0

NVMe disk cause kernel panic in installer
by bossman90＠gmail.com 21 Jul '19

21 Jul '19

Hey, I have problems with running oVirt installer on my H370M-ITX mobo. I have installed NVMe disk in it (which I want to use to install oVirt there) and installer crashes with kernel panic: https://photos.app.goo.gl/AY9T93bEs8sCQzRF7 Is there any issues with compatibility oVirt and NVMe?

3 6

Re: NVMe disk cause kernel panic in installer
by Strahil 20 Jul '19

20 Jul '19

Then, you can try a newer version of the kernel from epel repo. Also, check in the CentOS forum - maybe someone had a similar issue. Best Regards, Strahil NikolovOn Jul 20, 2019 03:27, bossman90(a)gmail.com wrote: > > Unfortunately, it didn't help. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2YSRWPT536PAE…

1 0

What values should i give in this ovirt api to get the correct response
by midhun raj 20 Jul '19

20 Jul '19

I am following this link. https://ovirt.org/documentation/admin-guide/chap-Backups_and_Migration.html… I got response for GET /api/vms/4ef8b995-ae58-49d6-8980-ef0fa6688d92/snapshots/1b94ca4e-8c03-4166-8d29-2b831e24b90e/disks HTTP/1.1 Accept: application/xml Content-type: application/xml Identify disk id "Array ( [disk] => Array ( [0] => Array ( [actual_size] => 0 [alias] => GlanceDisk-8700235 [format] => cow [image_id] => a42bda58-7556-4ebe-a54d-03dc0a3b1f54 [propagate_errors] => false [provisioned_size] => 8589934592 [read_only] => false [shareable] => false [sparse] => true [status] => ok [storage_type] => image [wipe_after_delete] => false [snapshot] => Array ( [id] => 1b94ca4e-8c03-4166-8d29-2b831e24b90e ) [storage_domains] => Array ( [storage_domain] => Array ( [0] => Array ( [id] => ad6aea6d-ff90-457f-98eb-99380949e23a ) ) ) [name] => GlanceDisk-8700235 [description] => CentOS 7 Generic Cloud Image v1711 for x86_64 (8700235) [id] => dded245b-4a73-48b4-a1b4-ba98c37490ad ) What is the values that i have to give to the POST request in this api POST /api/vms/22222222-2222-2222-2222-222222222222/disks/ HTTP/1.1 Accept: application/xml Content-type: application/xml <disk id="11111111-1111-1111-1111-111111111111"> <snapshot id="11111111-1111-1111-1111-111111111111"/> <active>true</active> </disk> to replace 1)vms ===>> 22222222-2222-2222-2222-222222222222 2)disk id 11111111-1111-1111-1111-111111111111 3)snapshot id 11111111-1111-1111-1111-111111111111

1 0

Re: Storage domain 'Inactive' but still functional
by Martijn Grendelman 19 Jul '19

19 Jul '19

Hi! Thanks. Like I wrote, I have metadata backups from /etc/lvm/backup and -/archive, and I also have the current metadata as it exists on disk. What I'm most concerned about, is the proposed procedure. I would create a backup of the VG, but I'm not sure what would be the most sensible way to do it. I could make a new iSCSI target and simply 'dd' the whole disk over, but that would take quite some time (it's 2,5 TB) and there are VMs that can't really be down for that long. And I'm not even sure that dd'ing the disk like that is a sensible strategy. Moving disks out of the domain is currently not possible. oVirt says 'Source Storage Domain is not active'. Thanks, Martijn. Op 18-7-2019 om 17:44 schreef Strahil Nikolov: Can you check the /etc/lvm/backup and /etc/lvm/archive on your SPM host (check the other hosts, just in case you find anything useful) ? Usually LVM makes backup of everything. I would recommend you to: 1. Create a backup of the problematic VG 2. Compare the backup file and a file from backup/archive folders for the same VG Check what is different with diff/vimdiff . It might give you a clue. I had some issues (non-related to oVirt) and restoring the VG from older backup did help me .Still ,any operation on block devices should be considered risky and a proper backup is needed. You could try to move a less important VM's disks out of this storage domain to another one. If it succeeds - then you can evacuate all VMs away before you can start "breaking" the storage domain. Best Regards, Strahil Nikolov В четвъртък, 18 юли 2019 г., 16:59:46 ч. Гринуич+3, Martijn Grendelman <martijn.grendelman(a)isaac.nl><mailto:martijn.grendelman@isaac.nl> написа: Hi, It appears that O365 has trouble delivering mails to this list, so two earlier mails of mine are still somewhere in a queue and may yet be delivered. This mail has all of the content of 3 successive mails. I apologize for this format. Op 18-7-2019 om 11:20 schreef Martijn Grendelman: Op 18-7-2019 om 10:16 schreef Martijn Grendelman: Hi, For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I need some help. Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a disk image off of it, to another domain. This failed, and somewhere in the process, the whole storage domain went to status 'Inactive'. From engine.log: 2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'. 2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2019-07-17 16:30:35,410+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) was deactivated by system because it's not visible by any of the hosts. The thing is, the domain is still functional on all my hosts. It carries over 50 disks, and all involved VMs are up and running, and don't seem to have any problems. Also, 'iscsiadm' on all hosts seems to indiciate that everything is fine with this specific target and reading from the device with dd, or getting its size with 'blockdev' all works without issue. When I try to reactivate the domain, these errors are logged: 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',), code = 358 2019-07-18 09:34:53,648+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) by martijn@-authz On the SPM host, there are errors that indicate problems with the LVM volume group: 2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429', spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48) 2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool] sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78 (sp:1125) 2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=[' /dev/mapper/23536316636393463: Checksum error at offset 2748693688832', " Couldn't read volume group metadata from /dev/mapper/23536316636393463.", ' Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/23536316636393463', ' Failed to scan VG from /dev/mapper/23536316636393463', ' Volume group "875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted: "Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83) I need help getting this storage domain back online. Can anyone here help me? If you need any additional information, please let me know! It appears the VG metadata is corrupt: /dev/mapper/23536316636393463: Checksum error at offset 2748693688832 Couldn't read volume group metadata from /dev/mapper/23536316636393463. Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/23536316636393463 Failed to scan VG from /dev/mapper/23536316636393463 Is this fixable? If so, how? So, I have found some information online, that suggests that PV metadata can be fixed by recreating the PV label using the correct PVID and a backup of the LVM metadata, like so: pvcreate -u <pv_uuid> --restorefile <lvm_metadata_backup> /dev/mapper/23536316636393463 Now I have the following two files: * An LVM metadata backup of yesterday 10:35, about 6 hours before the problem occurred. * The actual metadata found on the PV at offset 2748693688832 (obtained with hexedit on the block device). These are largely the same, but there are differences: * seqno = 1854 in the backup and 1865 in the actual metadata. * 3 logical volumes that are not present in the backup, but are in the actual metadata. I suspect that these are related to snapshots that were created for live storage migration, but I am not sure. In any case, I did NOT create any new disk images on this domain, so that can't be it. Now, support I wanted to try the 'pvcreate' route, then: * what would be the chances of success? Is this procedure at all advisable, or is there an alternative? * which restore file (1854 or 1865) should I use for the restore? * can I do this while the VG is in use? I tried running the command without --force, and it said 'Can't open /dev/mapper/23536316636393463 exclusively. Mounted filesystem?'. I didn't dare to try it with '--force'. I could really use some advice on how to proceed. This are about 36 VMs that have one or more disks on this domain. I could bring them down, although doing so for extended amounts of time would be problematic. I want to be careful, obviously, especially since the actual storage doesn't seem to be impacted at this time. The VMs are all still running without issue, and if I'm about to embark on a dangerous journey that could cause data loss, I need a contingency / recovery plan. Hoping someone can help... Best regards, Martijn Grendelman _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKWOTMY6MKDEQ… -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend.

2 1