January 2020 - Devel - oVirt List Archives

OST CI: Time issue?
by Yedidyah Bar David 20 Jan '20

20 Jan '20

Hi all, I looked at failure of [1]. It timed out while waiting for the engine to be up, after restarting the engine VM. lago.log [2] has: 2020-01-20 04:16:06,026::008_restart_he_vm.py::_start_he_vm::173::root::INFO::Starting VM... ... 2020-01-20 04:16:09,823::008_restart_he_vm.py::_start_he_vm::178::root::INFO::Waiting for VM to be UP... ... 2020-01-20 04:16:25,113::008_restart_he_vm.py::_start_he_vm::186::root::INFO::VM is UP. 2020-01-20 04:16:25,113::008_restart_he_vm.py::_wait_for_engine_health::190::root::INFO::Waiting for engine to start... ... Then there is a loop of running 'hosted-engine --vm-status --json' and parsing the status, specifically checking the health. The health status does not change until the timeout (after 10 minutes), but after 'vm', there is 'detail', which does change. First one is 'Powering up': 2020-01-20 04:16:25,113::ssh.py::ssh::89::lago.ssh::DEBUG::Command 9ef791c0 on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5199 (Sun Jan 19 23:16:24 2020)\nhost-id=1\nscore=2400\nvm_conf_refresh_time=5199 (Sun Jan 19 23:16:24 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 2400, "stopped": false, "maintenance": false, "crc32": "e7f42f2a", "local_conf_timestamp": 5199, "host-ts": 5199}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5190 (Sun Jan 19 23:16:15 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5190 (Sun Jan 19 23:16:15 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3657e8b", "local_conf_timestamp": 5190, "host-ts": 5190}, "global_maintenance": true} ... Then a few similar ones, last one is from '2020-01-20 04:17:22,980', and then it changes to 'Up': 2020-01-20 04:17:27,517::ssh.py::ssh::89::lago.ssh::DEBUG::Command c426f53a on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5259 (Sun Jan 19 23:17:24 2020)\nhost-id=1\nscore=2928\nvm_conf_refresh_time=5259 (Sun Jan 19 23:17:24 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 2928, "stopped": false, "maintenance": false, "crc32": "08d4bfe1", "local_conf_timestamp": 5259, "host-ts": 5259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5250 (Sun Jan 19 23:17:16 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5251 (Sun Jan 19 23:17:16 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ab277249", "local_conf_timestamp": 5251, "host-ts": 5250}, "global_maintenance": true} /var/log/messages of the engine vm [3] has: Jan 19 23:17:57 lago-he-basic-role-remote-suite-4-3-engine kernel: Initializing cgroup subsys cpuset Meaning, the first line after the reboot has a timestamp that's almost 2 minutes after starting the VM. That's a long time. That's half a minute after detail changed to 'Up', above (no idea where we take this 'detail' from, though). Then, first line that is not from the kernel: Jan 19 23:17:58 lago-he-basic-role-remote-suite-4-3-engine systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) then it starts lots of services, and then: Jan 19 23:20:31 lago-he-basic-role-remote-suite-4-3-engine systemd: Started oVirt Engine websockets proxy. Jan 19 23:20:33 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 45.33.103.94 Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: System clock wrong by 3.919225 seconds, adjustment started Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine systemd: Time has been changed Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: System clock was stepped by 3.919225 seconds Jan 19 23:20:38 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 129.250.35.250 Jan 19 23:20:43 lago-he-basic-role-remote-suite-4-3-engine cloud-init: Cloud-init v. 18.5 running 'modules:config' at Mon, 20 Jan 2020 04:20:39 +0000. Up 182.82 seconds. and later: Jan 19 23:21:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Started Update UTMP about System Runlevel Changes. Jan 19 23:21:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Startup finished in 22.344s (kernel) + 32.237s (initrd) + 3min 815ms (userspace) = 3min 55.397s. Jan 19 23:21:43 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 206.55.191.142 Jan 19 23:23:52 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 129.250.35.250 Jan 19 23:26:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Created slice User Slice of root. engine.log's [4] first line after the reboot is: 2020-01-19 23:26:18,251-05 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 103) [] Loaded file '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf'. which is a mere 40 seconds before the timeout, and server.log's [5] first line is: 2020-01-19 23:23:13,392-05 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "apidoc.war" (runtime-name: "apidoc.war") which is a 3 minutes before that. I do not know much about what the engine does during its startup, and how the logs should look, and how long it should take between startup and a working health page. If this all looks normal, then perhaps we should simply give it more than 10 minutes. Otherwise, there is either heavy load on the infra, or perhaps some NTP problems. Anyone has a clue? Anyway, now pushed a patch [6] to allow up to 20 minutes, at least to reduce the noise. Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [3] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [5] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [6] https://gerrit.ovirt.org/106411 -- Didi

1 0

CI: Multiple jobs hang on "Archiving Artifacts"
by Amit Bawer 19 Jan '20

19 Jan '20

For the last 40 minutes, multiple CI jobs are hanging on "Archiving Artifacts" stage: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/ could be a false alarm, but none seem to progress.

2 2

maven central brings some surprises
by Fedor Gavrilov 17 Jan '20

17 Jan '20

Hi, Maven central seems to be not in the mood today for me: [ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:2.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-jar-plugin:jar:2.2: Could not transfer artifact org.apache.maven.plugins:maven-jar-plugin:pom:2.2 from/to central (http://repo1.maven.org/maven2) Failed to transfer file: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2…. Return code is: 501 , ReasonPhrase:HTTPS Required. -> [Help 1] Does anyone know if that's somehow a proxy thing or something else? I'm asking because I had a joy of dealing with lying proxy before. Also, iirc jcenter had a reputation of being more reliable, can we use it instead somehow? Thanks, Fedor

3 2

Restore fails whether engine is on or off
by raymond.francis＠oracle.com 16 Jan '20

16 Jan '20

Wondering whether this is an issue; I had an engine where i created a backup and stored elsewhere. This engine was reinstalled and the backup file (called file_name) used to restore below. However i get 2 different errors based on whether engine is on.... [root@dub-mgrfarm113 ~]# service ovirt-engine status Redirecting to /bin/systemctl status ovirt-engine.service ● ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2020-01-16 16:13:41 GMT; 2h 5min ago Main PID: 5621 (ovirt-engine.py) CGroup: /system.slice/ovirt-engine.service ├─5621 /usr/bin/python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify st... └─5661 ovirt-engine -server -XX:+TieredCompilation -Xms1991M -Xmx1991M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.clien... Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com systemd[1]: Starting oVirt Engine... Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16 16:13:40,816+0000 ovirt-engine: INFO _detectJBossV...l=36 Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16 16:13:41,795+0000 ovirt-engine: INFO _detectJBossV...'[]' Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com systemd[1]: Started oVirt Engine. Hint: Some lines were ellipsized, use -l to show in full. [root@dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: file_name log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181932.log Preparing to restore: FATAL: Engine service is active - can not restore backup or off....... [root@dub-mgrfarm113 ~]# service ovirt-engine stop Redirecting to /bin/systemctl stop ovirt-engine.service [root@dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: file_name log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181952.log Preparing to restore: - Unpacking file 'file_name' Restoring: - Files FATAL: Can't connect to database 'engine'. Please see '/usr/bin/engine-backup --help'. In addition, possibly related, the web page for the server will no longer launch even when ovirt-engine and httpd services are running while. Any troubleshooting tips or has anyone seen same before?

1 0

FYI for Intellij IDEA & CentOS 7 users
by Fedor Gavrilov 16 Jan '20

16 Jan '20

Hi, Just wanted to share in case someone will have same issue as me. Recent version of IDEA requires new libgcc which is not available for CentOS 7, which has 2.17: Exception in thread "JavaFX Application Thread" java.lang.UnsatisfiedLinkError: /home/fgavrilo/Software/idea-IC-193.5662.53/jbr/lib/libjfxmedia.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found The simplest solution to this is probably using flatpak distribution: sudo yum install flatpak flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo flatpak install flathub com.jetbrains.IntelliJ-IDEA-Community flatpak run com.jetbrains.IntelliJ-IDEA-Community Fedor

1 0

REST and Python APIs -- jobs and disk updates
by Martin Kletzander 16 Jan '20

16 Jan '20

Hello everyone, so I am trying to do some work related to oVirt and I am handling disks using the ovirtsdk4 python API. However I am not sure about how to do two things. First one is about avoiding an active wait or at least minimizing the guesswork. When creating a new disk I have to then wait in a loop until it is ready, polling it and checking if it is in 'OK' status. However the creation returns a job which has way better information on what is happening (is it still running? did it fail?) and I was wondering if there is a way how to get the job information using the Python SDK. Another thing is that, apparently, updating a disk only works for updating the qcow_version and nothing else. And I need to update its shareable attribute, which is completely possible using the web interface. What is the proper way to do that using REST APIs or the Python SDK? Thanks for any info, Martin

3 3

How to create IP address for a VM in Ovirt
by raymond.francis＠oracle.com 16 Jan '20

16 Jan '20

How can i configure an IP address on a NIC interface for a VM. Need to set it to DHCP to do it but im not spotting the option?

1 0

Please review https://github.com/oVirt/ovirt-ansible-shutdown-env/pulls
by Sandro Bonazzola 15 Jan '20

15 Jan '20

Please review https://github.com/oVirt/ovirt-ansible-shutdown-env/pulls Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: [PATCH 0/4][RFC] Drop media update support in virDomainAttachDevice*
by Daniel P. Berrangé 15 Jan '20

15 Jan '20

On Wed, Jan 15, 2020 at 04:24:54PM +0800, Han Han wrote: > https://bugzilla.redhat.com/show_bug.cgi?id=1788793 > > Before virDomainUpdateDeviceFlags, virDomainAttachDevice* were used for > media update. Now we have introduced virDomainUpdateDeviceFlags for 9 > years(since commit 46a2ea3689) but keep that compatibility of > virDomainAttachDevice* until now. > > I think it is time to remove the compatibility since it doesn't work as > well as virDomainUpdateDeviceFlags on disk xml validation. > > I wonder if any uplayer software relying on this compatibility... NACK, this is explicitly the kind of thing we will never do. Providing a long term stable API means keeping existing methods operating unchanged, even if we have since introduced a new method which can do the same thing in a better way. The cost of keeping this existing code is negligible in context to the maint work of libvirt in general. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

1 0

planned Jenkins restart
by Evgheni Dereveanchin 14 Jan '20

14 Jan '20

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is back online. -- Regards, Evgheni Dereveanchin

1 1

OST manual suite fails for more than a month
by Tal Nisan 14 Jan '20

14 Jan '20

Ran it myself now on master just to have a baseline [1], it fails with: *17:34:58* # verify_add_all_hosts: ERROR (in 0:20:07) <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…>*17:34:58* @ Run test: 002_bootstrap.py: ERROR (in 0:27:56) <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…>*17:34:58* Error occured, aborting Who can have a look? [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…

3 2

[ ERROR ] Failed to execute stage 'Setup validation': Aborted by user
by raymond.francis＠oracle.com 14 Jan '20

14 Jan '20

Not seen this message googling, im the only user too. Not sure whats up here engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213536-k7u9m1.log Version: otopi-1.8.3 (otopi-1.8.3-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization Do you want to remove all components? (Yes, No) [Yes]: --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213536-k7u9m1.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200113213548-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed [root@dub-mgrfarm113 ~]# engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213556-ao8uv5.log Version: otopi-1.8.3 (otopi-1.8.3-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization Do you want to remove all components? (Yes, No) [Yes]: --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213556-ao8uv5.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200113213601-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed

2 1

OST: Is it still possible to run it without mock?
by Milan Zamazal 13 Jan '20

13 Jan '20

Hi, when trying to run basic-suite-master using run_suite.sh on my machine, I get the following error: /home/pdm/ovirt-system-tests/run_suite.sh: line 532: /var/lib/ci_toolbox/safe_download.sh: No such file or directory ci_toolbox/safe_download.sh is in Jenkins repo -- is mock now required to run OST now or is it just a mistake (either in OST or on my side)? Thanks, Milan

2 4

Cannot delete disk despite all disk permissions given to user
by raymond.francis＠oracle.com 13 Jan '20

13 Jan '20

Possible bug i found in testing (or could be my own understanding). If i give a user the the following capabiities, wouldnt it stand to reason that he can then delete a disk? StorageAdmin DiskCreator DiskProfileUser SuperUser DiskOperator I get a permissions issue if i then try to delete a disk. Let me know if any logs etc needed. Thanks Raymond

1 1

openstack glace images credentials
by Dana Elfassy 12 Jan '20

12 Jan '20

Hi, I'm trying to get the credentials for several images on ovirt-image-repository (OpenStack Glance) on storage domains in the webadmin Who can help me with this? Thanks, Dana

3 3

oVirt and Gluster Hyperconverged - VM's Bad volume specification
by thomas.rockey＠datasphere.com 09 Jan '20

09 Jan '20

I am new to this forum so I apologize before hand if I don’t present the right content you are looking for or miss content you need. Background: By no means am I an expert with Ovirt and glusterfs. That said I have been using, managing, building out Ovirt (single hosts) and oVirt with Gluster Hyperconverged environments for 5 years or more. I started building out Ovirt environments with oVirt Engine Version: 3.6.7.5-1.el6 and earlier and now I’m using the latest oVirt with Gluster Hyperconverged. Current hardware and software layout: For the last 8 months I have been using a oVirt with Gluster Hyperconverged to host in total about 100 VM’s. My hardware layout in one environment is 5 Dell R410 3 of them configured with Gluster Hyperconverged and the other 2 are just added hosts. Below is a detailed list! Manufacturer: Dell Inc. PowerEdge R410 CPU Model Name: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz CPU Cores per Socket: 4 CPU Type: Intel Westmere IBRS SSBD Family Dell PERC H700 4 SAS Seagate 4 TB drives 7.2k 2 one gig links – NIC 1 for frontend and NIC 2 for gluster backend My software layout is: OS Version: RHEL - 7 - 7.1908.0.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 1062.9.1.el7.x86_64 KVM Version: 2.12.0 - 33.1.el7_7.4 LIBVIRT Version: libvirt-4.5.0-23.el7_7.3 VDSM Version: vdsm-4.30.38-1.el7 SPICE Version: 0.14.0 - 7.el7 GlusterFS Version: glusterfs-6.6-1.el7 CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.11.0-4.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 VNC Encryption: Disabled My network layout is: 3 HP 3800-48G-4SFP+ Switch (J9576A) running FULL MESH Issue/timeline: • All 3 of the HP 3800 were rebooted at the same time and were down for 5 to 10 seconds before they came back up (meaning pingable and responsive). • A little more than 85% (36 or so) of the VM’s I had running all went into a pause state, do to and unknow storage error. • The gluster volume heal state went all the way up to 2300 on vmstore (OS data location) • After heal completed on the vmstore (took about an hour) 85% of the VM’s failed to launch with an error (see below). VM broadsort is down with error. Exit message: Bad volume specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'b1bf3f56-a453-4383-a350-288bee06445b', 'index': 0, 'iface': 'virtio', 'apparentsize': '274877906944', 'specParams': {}, 'cache': 'none', 'imageID': 'b1bf3f56-a453-4383-a350-288bee06445b', 'truesize': '106767498240', 'type': 'disk', 'domainID': 'a7119613-a5ba-4a97-802b-0a985c647381', 'reqsize': '0', 'format': 'raw', 'poolID': '699fd2d6-c461-11e9-8b83-00163e18a045', 'device': 'disk', 'path': 'vmstore/a7119613-a5ba-4a97-802b-0a985c647381/images/b1bf3f56-a453-4383-a350-288bee06445b/25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': '25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'diskType': 'network', 'alias': 'ua-b1bf3f56-a453-4383-a350-288bee06445b', 'hosts': [{'name': 'glust01.mydomain.local', 'port': '0'}], 'discard': False}. Everyone of the VM’s had this same error and I had to find backups and old images to bring them back online. I deleted some of the corrupted VM’s that I had current images of, to get them back up. You shouldn’t be afraid to reboot 1, 2, 3, or even all of your switches at once because of a human error, power outage, or a simple update. Then have to worry about your VM’s getting corrupted because of this concerns me greatly. I am now thinking that I didn't setup oVirt with Gluster Hyperconverged correctly because of this issue. Have I missed something in the documentation or network layout/setup that would prevent this from happening again? I have searched the web for a few days now trying to find threads related to my situation with no luck. I want to thank you for your time and it is greatly appreciated!

1 0

oVirt and Gluster Hyperconverged - VM's Bad volume specification
by thomas.rockey＠datasphere.com 09 Jan '20

09 Jan '20

I am new to this forum so I apologize before hand if I don’t present right content correctly or miss the content you need. Background: By no means am I an expert with Ovirt and glusterfs. That said I have been using, managing, building out Ovirt (single hosts) and oVirt with Gluster Hyperconverged environments for 5 years or more. I started building out Ovirt environments with oVirt Engine Version: 3.6.7.5-1.el6 and earlier and now I’m using the latest oVirt with Gluster Hyperconverged. Current hardware and software layout: For the last 8 months I have been using a oVirt with Gluster Hyperconverged to host in total about 100 VM’s. My hardware layout in one environment is 5 Dell R410 3 of them configured with Gluster Hyperconverged and the other 2 are just added hosts. Below is a detailed list! Manufacturer: Dell Inc. PowerEdge R410 CPU Model Name: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz CPU Cores per Socket: 4 CPU Type: Intel Westmere IBRS SSBD Family Dell PERC H700 4 SAS Seagate 4 TB drives 7.2k 2 one gig links – NIC 1 for frontend and NIC 2 for gluster backend My software layout is: OS Version: RHEL - 7 - 7.1908.0.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 1062.9.1.el7.x86_64 KVM Version: 2.12.0 - 33.1.el7_7.4 LIBVIRT Version: libvirt-4.5.0-23.el7_7.3 VDSM Version: vdsm-4.30.38-1.el7 SPICE Version: 0.14.0 - 7.el7 GlusterFS Version: glusterfs-6.6-1.el7 CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.11.0-4.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 VNC Encryption: Disabled My network layout is: 3 HP 3800-48G-4SFP+ Switch (J9576A) running FULL MESH Issue/timeline: • All 3 of the HP 3800 were rebooted at the same time and were down for 5 to 10 seconds before they came back up (meaning pingable and responsive). • A little more than 85% (36 or so) of the VM’s I had running all went into a pause state, do to and unknow storage error. • The gluster volume heal state went all the way up to 2300 on vmstore (OS data location) • After heal completed on the vmstore (took about an hour) 85% of the VM’s failed to launch with an error (see below). VM broadsort is down with error. Exit message: Bad volume specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'b1bf3f56-a453-4383-a350-288bee06445b', 'index': 0, 'iface': 'virtio', 'apparentsize': '274877906944', 'specParams': {}, 'cache': 'none', 'imageID': 'b1bf3f56-a453-4383-a350-288bee06445b', 'truesize': '106767498240', 'type': 'disk', 'domainID': 'a7119613-a5ba-4a97-802b-0a985c647381', 'reqsize': '0', 'format': 'raw', 'poolID': '699fd2d6-c461-11e9-8b83-00163e18a045', 'device': 'disk', 'path': 'vmstore/a7119613-a5ba-4a97-802b-0a985c647381/images/b1bf3f56-a453-4383-a350-288bee06445b/25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': '25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'diskType': 'network', 'alias': 'ua-b1bf3f56-a453-4383-a350-288bee06445b', 'hosts': [{'name': 'glust01.mydomain.local', 'port': '0'}], 'discard': False}. Everyone of the VM’s had this same error and I had to find backups and old images to bring them back online. I deleted some of the VM’s that I had current images of to get them back up. You shouldn’t be afraid to reboot 1, 2, 3, or even all of your switches at once because of a human error, power outage, or a simple update. Then have to worry about your VM’s getting corrupted because of this concerns me greatly that I didn’t setup oVirt with Gluster Hyperconverged correctly. Have I missed something in the documentation or network layout that would prevent this for happing? I want to thank you for your time and it is greatly appreciated!

1 0

Failed to add host using ansible runner
by Eyal Shenitzky 09 Jan '20

09 Jan '20

Hi, I am failing to add a new host to my env with the following message: 2019-12-25 10:57:17,587+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-63) [41ec72c1-88e2-402b-8bb9-f38c678d0bf0] EVENT_ID: VDS_INSTALL_FAILED(505), Host 10.35.0.158 installation failed. Failed to execute Ansible host-deploy role: null. Please check logs for more details: /home/engine/ovirt-engine/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20191225105714-10.35.0.158-41ec72c1-88e2-402b-8bb9-f38c678d0bf0.log. The host is Fedora-30 Ansible version is - 2.9.1 Ansible runner version - 1.3.4 /etc/ansible-runner-service/config.yaml: --- version: 1 target_user: root playbooks_root_dir: '/home/engine/ovirt-engine/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '/home/engine/ovirt-engine/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root There are no logs at all at the specified location in the error message. Did someone encounter that issue? Thanks, -- Regards, Eyal Shenitzky

6 13

New dependency for development environment
by Ondra Machacek 09 Jan '20

09 Jan '20

Hello, we are going to merge a series of patches to master branch, which integrates ansible-runner with oVirt engine. When the patches will be merged you will need to install new package called ansible-runner- service-dev, and follow instructions so your dev-env will keep working smoothly(all relevant info will be also in README.adoc): 1) sudo dnf update ovirt-release-master 2) sudo dnf install -y ansible-runner-service-dev 3) Edit `/etc/ansible-runner-service/config.yaml` file as follows: --- playbooks_root_dir: '$PREFIX/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '$PREFIX/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root Where `$PREFIX` is the prefix of your development environment prefix, which you've specified during the compilation of the engine. 4) Restart and enable ansible-runner-service: # systemctl restart ansible-runner-service # systemctl enable ansible-runner-service That's it, your dev-env should start using the ansible-runner-service for host-deployment etc. Please note that only Fedora 30/31 and Centos7 was packaged, and are natively supported! Thanks, Ondra

9 24

Failed to install ovirt-imageio-proxy for fc30
by Shani Leviim 08 Jan '20

08 Jan '20

Hi All, I ran into a dependencies issue while trying to install imageio over fc30. I currently have a ovirt-imageio-common-1.6.2-0.202001051603.git77cf282.fc30.x86_64 but when I'm trying to install ovirt-imageio-proxy, I'm getting the following error: Error: Problem: conflicting requests - nothing provides ovirt-imageio-common = 1.5.0 needed by ovirt-imageio-proxy-1.5.0-0.201812112104.gitf7d8234.fc30.noarch I've also noticed that the available version in the repository for fc30 is ovirt-imageio-proxy-1.5.0 [1] While version 1.6.2 is available for rhel7 [2]. Please assist. [1] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc30/noarch/ovirt… [2] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ovirt-… *Regards,* *Shani Leviim*

2 2

ovirt-engine has been tagged (ovirt-engine-4.3.8.2)
by Tal Nisan 07 Jan '20

07 Jan '20

1 0

OST is failing - Last successful run was Dec-13-2019
by Amit Bawer 06 Jan '20

06 Jan '20

Seems we have NFS permissions issue for el8 vdsm in some of the runs. Example from https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/arti… : 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1 (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)... 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata (monitor:501) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 499, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\': Operation not permitted\n")) 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] Calling 'StoragePool.disconnect' in bridge with {'storagepoolID': 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey': 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (__init__:329) 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439', hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786, flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48) Probably need to set NFS server export options as in https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7

2 2

Unexpected exception when trying to add a storage domain
by Dana Elfassy 05 Jan '20

05 Jan '20

Hi, When trying to add a storage domain to a 4.4 host I'm getting this error message: Error while executing action New NFS Storage Domain: Unexpected exception The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') 2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () 2020-01-02 09:38:33,583-0500 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') aborting: Task is aborted: 'value=Could not initialize cluster lock: () abortedcode=701' (task:1184) 2020-01-02 09:38:33,584-0500 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83) 2020-01-02 09:38:33,584-0500 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 701) in 18.70 seconds (__init__:312) 2020-01-02 09:38:33,730-0500 INFO (jsonrpc/2) [vdsm.api] START disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'protocol_version': 'auto', 'port': '', 'iqn': '', 'connection': 'vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/nfs1', 'ipv6_enabled': 'false', 'id': 'bf075967-060b-4e60-9b0b-bb170fe073f9', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:192.168.100.1,36452, flow_id=f4f79738-b361-4688-95c5-3454f52b505d, task_id=dd7266ed-69ec-4e5b-9216-850d9db8ea3b (api:48) Can someone help me with it? Thanks, Dana

2 4

Who's the owner of "vdsm_hooks/checkimages/before_vm_start.py"?
by Pavel Bar 03 Jan '20

03 Jan '20

Hi, Can someone please point me to a relevant person? It looks like there is a potential issue with "*getImageSize()*" function there: *'image_bytes' might be referenced before assignment*. There is also a suspicion that this code is not used at all, so instead of fixing it might be a good idea to just delete it. See the code below: def getImageSize(disk_image, driver_type): ''' Obtain qcow2 image size in GiBs ''' if driver_type == 'block': dev_buffer = ' ' * 8 with open(disk_image) as device: dev_buffer = fcntl.ioctl(device.fileno(), BLKGETSIZE64, dev_buffer) image_bytes = struct.unpack(FORMAT, dev_buffer)[0] elif driver_type == 'file': image_bytes = os.stat(disk_image).st_size return float(image_bytes / GIB) Thank you in advance! Pavel

2 1

OST Fails for missing glusterfs mirrors at host-deploy
by Amit Bawer 02 Jan '20

02 Jan '20

Snippet From: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6293/console 23:31:25 + cd /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/deployment-basic-suite-master 23:31:25 + lago ovirt deploy 23:31:26 @ Deploy oVirt environment: 23:31:26 # Deploy environment: 23:31:26 * [Thread-2] Deploy VM lago-basic-suite-master-host-0: 23:31:26 * [Thread-3] Deploy VM lago-basic-suite-master-host-1: 23:31:26 * [Thread-4] Deploy VM lago-basic-suite-master-engine: 23:32:15 * [Thread-3] Deploy VM lago-basic-suite-master-host-1: Success (in 0:00:49) 23:32:39 STDERR 23:32:39 + yum -y install ovirt-host 23:32:39 Error: Error downloading packages: 23:32:39 Cannot download glusterfs-6.6-1.el8.x86_64.rpm: All mirrors were tried 23:32:39 23:32:39 - STDERR 23:32:39 + yum -y install ovirt-host 23:32:39 Error: Error downloading packages: 23:32:39 Cannot download glusterfs-6.6-1.el8.x86_64.rpm: All mirrors were tried 23:32:39 23:32:39 * [Thread-2] Deploy VM lago-basic-suite-master-host-0: ERROR (in 0:01:13) 23:38:05 * [Thread-4] Deploy VM lago-basic-suite-master-engine: ERROR (in 0:06:39) 23:38:05 # Deploy environment: ERROR (in 0:06:39) 23:38:06 @ Deploy oVirt environment: ERROR (in 0:06:39) 23:38:06 Error occured, aborting 23:38:06 Traceback (most recent call last): 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 383, in do_run 23:38:06 self.cli_plugins[args.ovirtverb].do_run(args) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run 23:38:06 self._do_run(**vars(args)) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 573, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in wrapper 23:38:06 return func(*args, prefix=prefix, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 181, in do_deploy 23:38:06 prefix.deploy() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 127, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/prefix.py", line 284, in deploy 23:38:06 return super(OvirtPrefix, self).deploy() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in wrapped 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1671, in deploy 23:38:06 self.virt_env.get_vms().values() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 104, in invoke_in_parallel 23:38:06 return vt.join_all() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue 23:38:06 queue.put({'return': func()}) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1662, in _deploy_host 23:38:06 host.name(), 23:38:06 LagoDeployError: /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/deployment-basic-suite-master/default/scripts/_home_jenkins_agent_workspace_ovirt-system-tests_manual_ovirt-system-tests_basic-suite-master_deploy-scripts_setup_1st_host_el7.sh failed with status 1 on lago-basic-suite-master-host-0 23:38:06 + res=1 23:38:06 + cd - 23:38:06 /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests 23:38:06 + return 1 23:38:06 + env_collect /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy 23:38:06 + local tests_out_dir=/home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy 23:38:06 + [[ -e /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master ]] 23:38:06 + mkdir -p /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master 23:38:06 + cd /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/deployment-basic-suite-master/current 23:38:06 + lago collect --output /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy 23:38:08 @ Collect artifacts: 23:38:08 # [Thread-1] lago-basic-suite-master-host-0: 23:38:08 # [Thread-2] lago-basic-suite-master-host-1: 23:38:08 # [Thread-3] lago-basic-suite-master-engine: 23:38:10 # [Thread-1] lago-basic-suite-master-host-0: Success (in 0:00:02) 23:38:10 # [Thread-2] lago-basic-suite-master-host-1: Success (in 0:00:02) 23:38:16 # [Thread-3] lago-basic-suite-master-engine: Success (in 0:00:07) 23:38:16 @ Collect artifacts: Success (in 0:00:07) 23:38:16 + cp -a logs /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy/lago_logs 23:38:16 + cd - 23:38:16 /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests 23:38:16 + echo '@@@ ERROR: Failed in deploy stage' 23:38:16 @@@ ERROR: Failed in deploy stage

2 1