- Devel - oVirt List Archives

OST CI: Time issue?
by Yedidyah Bar David 20 Jan '20

20 Jan '20

Hi all, I looked at failure of [1]. It timed out while waiting for the engine to be up, after restarting the engine VM. lago.log [2] has: 2020-01-20 04:16:06,026::008_restart_he_vm.py::_start_he_vm::173::root::INFO::Starting VM... ... 2020-01-20 04:16:09,823::008_restart_he_vm.py::_start_he_vm::178::root::INFO::Waiting for VM to be UP... ... 2020-01-20 04:16:25,113::008_restart_he_vm.py::_start_he_vm::186::root::INFO::VM is UP. 2020-01-20 04:16:25,113::008_restart_he_vm.py::_wait_for_engine_health::190::root::INFO::Waiting for engine to start... ... Then there is a loop of running 'hosted-engine --vm-status --json' and parsing the status, specifically checking the health. The health status does not change until the timeout (after 10 minutes), but after 'vm', there is 'detail', which does change. First one is 'Powering up': 2020-01-20 04:16:25,113::ssh.py::ssh::89::lago.ssh::DEBUG::Command 9ef791c0 on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5199 (Sun Jan 19 23:16:24 2020)\nhost-id=1\nscore=2400\nvm_conf_refresh_time=5199 (Sun Jan 19 23:16:24 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 2400, "stopped": false, "maintenance": false, "crc32": "e7f42f2a", "local_conf_timestamp": 5199, "host-ts": 5199}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5190 (Sun Jan 19 23:16:15 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5190 (Sun Jan 19 23:16:15 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3657e8b", "local_conf_timestamp": 5190, "host-ts": 5190}, "global_maintenance": true} ... Then a few similar ones, last one is from '2020-01-20 04:17:22,980', and then it changes to 'Up': 2020-01-20 04:17:27,517::ssh.py::ssh::89::lago.ssh::DEBUG::Command c426f53a on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5259 (Sun Jan 19 23:17:24 2020)\nhost-id=1\nscore=2928\nvm_conf_refresh_time=5259 (Sun Jan 19 23:17:24 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 2928, "stopped": false, "maintenance": false, "crc32": "08d4bfe1", "local_conf_timestamp": 5259, "host-ts": 5259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5250 (Sun Jan 19 23:17:16 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5251 (Sun Jan 19 23:17:16 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ab277249", "local_conf_timestamp": 5251, "host-ts": 5250}, "global_maintenance": true} /var/log/messages of the engine vm [3] has: Jan 19 23:17:57 lago-he-basic-role-remote-suite-4-3-engine kernel: Initializing cgroup subsys cpuset Meaning, the first line after the reboot has a timestamp that's almost 2 minutes after starting the VM. That's a long time. That's half a minute after detail changed to 'Up', above (no idea where we take this 'detail' from, though). Then, first line that is not from the kernel: Jan 19 23:17:58 lago-he-basic-role-remote-suite-4-3-engine systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) then it starts lots of services, and then: Jan 19 23:20:31 lago-he-basic-role-remote-suite-4-3-engine systemd: Started oVirt Engine websockets proxy. Jan 19 23:20:33 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 45.33.103.94 Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: System clock wrong by 3.919225 seconds, adjustment started Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine systemd: Time has been changed Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: System clock was stepped by 3.919225 seconds Jan 19 23:20:38 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 129.250.35.250 Jan 19 23:20:43 lago-he-basic-role-remote-suite-4-3-engine cloud-init: Cloud-init v. 18.5 running 'modules:config' at Mon, 20 Jan 2020 04:20:39 +0000. Up 182.82 seconds. and later: Jan 19 23:21:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Started Update UTMP about System Runlevel Changes. Jan 19 23:21:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Startup finished in 22.344s (kernel) + 32.237s (initrd) + 3min 815ms (userspace) = 3min 55.397s. Jan 19 23:21:43 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 206.55.191.142 Jan 19 23:23:52 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 129.250.35.250 Jan 19 23:26:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Created slice User Slice of root. engine.log's [4] first line after the reboot is: 2020-01-19 23:26:18,251-05 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 103) [] Loaded file '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf'. which is a mere 40 seconds before the timeout, and server.log's [5] first line is: 2020-01-19 23:23:13,392-05 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "apidoc.war" (runtime-name: "apidoc.war") which is a 3 minutes before that. I do not know much about what the engine does during its startup, and how the logs should look, and how long it should take between startup and a working health page. If this all looks normal, then perhaps we should simply give it more than 10 minutes. Otherwise, there is either heavy load on the infra, or perhaps some NTP problems. Anyone has a clue? Anyway, now pushed a patch [6] to allow up to 20 minutes, at least to reduce the noise. Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [3] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [5] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [6] https://gerrit.ovirt.org/106411 -- Didi

1 0

CI: Multiple jobs hang on "Archiving Artifacts"
by Amit Bawer 19 Jan '20

19 Jan '20

For the last 40 minutes, multiple CI jobs are hanging on "Archiving Artifacts" stage: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/ could be a false alarm, but none seem to progress.

2 2

maven central brings some surprises
by Fedor Gavrilov 17 Jan '20

17 Jan '20

Hi, Maven central seems to be not in the mood today for me: [ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:2.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-jar-plugin:jar:2.2: Could not transfer artifact org.apache.maven.plugins:maven-jar-plugin:pom:2.2 from/to central (http://repo1.maven.org/maven2) Failed to transfer file: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2…. Return code is: 501 , ReasonPhrase:HTTPS Required. -> [Help 1] Does anyone know if that's somehow a proxy thing or something else? I'm asking because I had a joy of dealing with lying proxy before. Also, iirc jcenter had a reputation of being more reliable, can we use it instead somehow? Thanks, Fedor

3 2

Restore fails whether engine is on or off
by raymond.francis＠oracle.com 16 Jan '20

16 Jan '20

Wondering whether this is an issue; I had an engine where i created a backup and stored elsewhere. This engine was reinstalled and the backup file (called file_name) used to restore below. However i get 2 different errors based on whether engine is on.... [root@dub-mgrfarm113 ~]# service ovirt-engine status Redirecting to /bin/systemctl status ovirt-engine.service ● ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2020-01-16 16:13:41 GMT; 2h 5min ago Main PID: 5621 (ovirt-engine.py) CGroup: /system.slice/ovirt-engine.service ├─5621 /usr/bin/python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify st... └─5661 ovirt-engine -server -XX:+TieredCompilation -Xms1991M -Xmx1991M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.clien... Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com systemd[1]: Starting oVirt Engine... Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16 16:13:40,816+0000 ovirt-engine: INFO _detectJBossV...l=36 Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16 16:13:41,795+0000 ovirt-engine: INFO _detectJBossV...'[]' Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com systemd[1]: Started oVirt Engine. Hint: Some lines were ellipsized, use -l to show in full. [root@dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: file_name log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181932.log Preparing to restore: FATAL: Engine service is active - can not restore backup or off....... [root@dub-mgrfarm113 ~]# service ovirt-engine stop Redirecting to /bin/systemctl stop ovirt-engine.service [root@dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: file_name log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181952.log Preparing to restore: - Unpacking file 'file_name' Restoring: - Files FATAL: Can't connect to database 'engine'. Please see '/usr/bin/engine-backup --help'. In addition, possibly related, the web page for the server will no longer launch even when ovirt-engine and httpd services are running while. Any troubleshooting tips or has anyone seen same before?

1 0

FYI for Intellij IDEA & CentOS 7 users
by Fedor Gavrilov 16 Jan '20

16 Jan '20

Hi, Just wanted to share in case someone will have same issue as me. Recent version of IDEA requires new libgcc which is not available for CentOS 7, which has 2.17: Exception in thread "JavaFX Application Thread" java.lang.UnsatisfiedLinkError: /home/fgavrilo/Software/idea-IC-193.5662.53/jbr/lib/libjfxmedia.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found The simplest solution to this is probably using flatpak distribution: sudo yum install flatpak flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo flatpak install flathub com.jetbrains.IntelliJ-IDEA-Community flatpak run com.jetbrains.IntelliJ-IDEA-Community Fedor

1 0

REST and Python APIs -- jobs and disk updates
by Martin Kletzander 16 Jan '20

16 Jan '20

Hello everyone, so I am trying to do some work related to oVirt and I am handling disks using the ovirtsdk4 python API. However I am not sure about how to do two things. First one is about avoiding an active wait or at least minimizing the guesswork. When creating a new disk I have to then wait in a loop until it is ready, polling it and checking if it is in 'OK' status. However the creation returns a job which has way better information on what is happening (is it still running? did it fail?) and I was wondering if there is a way how to get the job information using the Python SDK. Another thing is that, apparently, updating a disk only works for updating the qcow_version and nothing else. And I need to update its shareable attribute, which is completely possible using the web interface. What is the proper way to do that using REST APIs or the Python SDK? Thanks for any info, Martin

3 3

How to create IP address for a VM in Ovirt
by raymond.francis＠oracle.com 16 Jan '20

16 Jan '20

How can i configure an IP address on a NIC interface for a VM. Need to set it to DHCP to do it but im not spotting the option?

1 0

Please review https://github.com/oVirt/ovirt-ansible-shutdown-env/pulls
by Sandro Bonazzola 15 Jan '20

15 Jan '20

Please review https://github.com/oVirt/ovirt-ansible-shutdown-env/pulls Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: [PATCH 0/4][RFC] Drop media update support in virDomainAttachDevice*
by Daniel P. Berrangé 15 Jan '20

15 Jan '20

On Wed, Jan 15, 2020 at 04:24:54PM +0800, Han Han wrote: > https://bugzilla.redhat.com/show_bug.cgi?id=1788793 > > Before virDomainUpdateDeviceFlags, virDomainAttachDevice* were used for > media update. Now we have introduced virDomainUpdateDeviceFlags for 9 > years(since commit 46a2ea3689) but keep that compatibility of > virDomainAttachDevice* until now. > > I think it is time to remove the compatibility since it doesn't work as > well as virDomainUpdateDeviceFlags on disk xml validation. > > I wonder if any uplayer software relying on this compatibility... NACK, this is explicitly the kind of thing we will never do. Providing a long term stable API means keeping existing methods operating unchanged, even if we have since introduced a new method which can do the same thing in a better way. The cost of keeping this existing code is negligible in context to the maint work of libvirt in general. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

1 0

planned Jenkins restart
by Evgheni Dereveanchin 14 Jan '20

14 Jan '20

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is back online. -- Regards, Evgheni Dereveanchin

1 1