- Infra - oVirt List Archives

[JIRA] (OVIRT-2583) ovirt-engine failing on master on test check_update_host
by Dafna Ron (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2583?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2583: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > ovirt-engine failing on master on test check_update_host > --------------------------------------------------------- > > Key: OVIRT-2583 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2583 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > ansible: Remove duplication of vnc_tls option - https://gerrit.ovirt.org/#/c/95169/ > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11267 > Mail sent to dev > {noformat} > [dron@dron post-002_bootstrap.py]$ egrep 89d99b74-fa6c-4d1d-a261-fb7ca7fcada2 lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log > 2018-11-21 13:41:34,209-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] method: get, params: [ad768a3b-d44b-4fd4-b57a-9029fca470a0], timeElapsed: 9ms > 2018-11-21 13:41:34,224-05 INFO [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCheckCommand] (default task-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Running command: HostUpgradeCheckCommand internal: false. Entities affected : ID: ad768a3b-d44b-4fd4-b57a-9029fca470a0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN > 2018-11-21 13:41:34,259-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] method: get, params: [ad768a3b-d44b-4fd4-b57a-9029fca470a0], timeElapsed: 13ms > 2018-11-21 13:41:34,273-05 INFO [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCheckInternalCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Running command: HostUpgradeCheckInternalCommand internal: true. Entities affected : ID: ad768a3b-d44b-4fd4-b57a-9029fca470a0 Type: VDS > 2018-11-21 13:41:34,277-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] method: get, params: [ad768a3b-d44b-4fd4-b57a-9029fca470a0], timeElapsed: 4ms > 2018-11-21 13:41:34,281-05 DEBUG [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Inventory hosts: [lago-basic-suite-master-host-0] > 2018-11-21 13:41:34,282-05 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Executing Ansible command: ANSIBLE_STDOUT_CALLBACK=hostupgradeplugin [/usr/bin/ansible-playbook, --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config, --check, --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa, --inventory=/tmp/ansible-inventory2326258642232337463, /usr/share/ovirt-engine/playbooks/ovirt-host-upgrade.yml] [Logfile: null] > 2018-11-21 13:41:37,610-05 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Ansible playbook command has exited with value: 2 > 2018-11-21 13:41:37,611-05 ERROR [org.ovirt.engine.core.bll.host.HostUpgradeManager] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Failed to run check-update of host 'lago-basic-suite-master-host-0'. > 2018-11-21 13:41:37,611-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.HostUpdatesChecker] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Failed to check if updates are available for host 'lago-basic-suite-master-host-0' with error message 'Failed to run check-update of host 'lago-basic-suite-master-host-0'.' > 2018-11-21 13:41:37,616-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] EVENT_ID: HOST_AVAILABLE_UPDATES_FAILED(839), Failed to check for available updates on host lago-basic-suite-master-host-0 with message 'Failed to run check-update of host 'lago-basic-suite-master-host-0'.'. > [dron@dron post-002_bootstrap.py]$ > {noformat} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2583) ovirt-engine failing on master on test check_update_host
by Dafna Ron (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2583?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2583: ---------------------------------- The change causing this issue has been reverted. Tomas is now debugging the issue so I am closing this ticket > ovirt-engine failing on master on test check_update_host > --------------------------------------------------------- > > Key: OVIRT-2583 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2583 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > ansible: Remove duplication of vnc_tls option - https://gerrit.ovirt.org/#/c/95169/ > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11267 > Mail sent to dev > {noformat} > [dron@dron post-002_bootstrap.py]$ egrep 89d99b74-fa6c-4d1d-a261-fb7ca7fcada2 lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log > 2018-11-21 13:41:34,209-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] method: get, params: [ad768a3b-d44b-4fd4-b57a-9029fca470a0], timeElapsed: 9ms > 2018-11-21 13:41:34,224-05 INFO [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCheckCommand] (default task-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Running command: HostUpgradeCheckCommand internal: false. Entities affected : ID: ad768a3b-d44b-4fd4-b57a-9029fca470a0 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN > 2018-11-21 13:41:34,259-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] method: get, params: [ad768a3b-d44b-4fd4-b57a-9029fca470a0], timeElapsed: 13ms > 2018-11-21 13:41:34,273-05 INFO [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCheckInternalCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Running command: HostUpgradeCheckInternalCommand internal: true. Entities affected : ID: ad768a3b-d44b-4fd4-b57a-9029fca470a0 Type: VDS > 2018-11-21 13:41:34,277-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] method: get, params: [ad768a3b-d44b-4fd4-b57a-9029fca470a0], timeElapsed: 4ms > 2018-11-21 13:41:34,281-05 DEBUG [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Inventory hosts: [lago-basic-suite-master-host-0] > 2018-11-21 13:41:34,282-05 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Executing Ansible command: ANSIBLE_STDOUT_CALLBACK=hostupgradeplugin [/usr/bin/ansible-playbook, --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config, --check, --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa, --inventory=/tmp/ansible-inventory2326258642232337463, /usr/share/ovirt-engine/playbooks/ovirt-host-upgrade.yml] [Logfile: null] > 2018-11-21 13:41:37,610-05 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Ansible playbook command has exited with value: 2 > 2018-11-21 13:41:37,611-05 ERROR [org.ovirt.engine.core.bll.host.HostUpgradeManager] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Failed to run check-update of host 'lago-basic-suite-master-host-0'. > 2018-11-21 13:41:37,611-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.HostUpdatesChecker] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] Failed to check if updates are available for host 'lago-basic-suite-master-host-0' with error message 'Failed to run check-update of host 'lago-basic-suite-master-host-0'.' > 2018-11-21 13:41:37,616-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [89d99b74-fa6c-4d1d-a261-fb7ca7fcada2] EVENT_ID: HOST_AVAILABLE_UPDATES_FAILED(839), Failed to check for available updates on host lago-basic-suite-master-host-0 with message 'Failed to run check-update of host 'lago-basic-suite-master-host-0'.'. > [dron@dron post-002_bootstrap.py]$ > {noformat} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2597) Two failures on reposync for tested repo
by Dafna Ron (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2597?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2597: ----------------------------- Resolution: Fixed Status: Done (was: To Do) > Two failures on reposync for tested repo > ---------------------------------------- > > Key: OVIRT-2597 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2597 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra > > We had two failures on master last night with > {noformat} > 23:45:59 [upgrade-from-release-suite.el7.x86_64] RuntimeError: Failed to run reposync 3 times for repoid: ovirt-4.2-tested-el7, aborting. > {noformat} > Full error: > https://pastebin.com/weBAa8ZD > Last failed build: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11707/consol… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2597) Two failures on reposync for tested repo
by Dafna Ron (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2597?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2597: ---------------------------------- I think this is due to the slowness we are experiencing from Jenkns > Two failures on reposync for tested repo > ---------------------------------------- > > Key: OVIRT-2597 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2597 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Labels: ost_failures, ost_infra > > We had two failures on master last night with > {noformat} > 23:45:59 [upgrade-from-release-suite.el7.x86_64] RuntimeError: Failed to run reposync 3 times for repoid: ovirt-4.2-tested-el7, aborting. > {noformat} > Full error: > https://pastebin.com/weBAa8ZD > Last failed build: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11707/consol… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- With regards to SSE-gateway I've found a very similar issue on their issue tracker to the one we're having: https://issues.jenkins-ci.org/browse/JENKINS-51057 People report reboots as a good workaround and I'm planning to do so in OVIRT-2606 so we should be good there. There's also a groovy script published to clean up the heap of SSE-gateway but it needs testing on staging. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- Here's the page linking directly to Jenkins: https://www.ovirt.org/node/#ovirt-node-master This is not the root cause of the major slowness so let's split it into a separate ticket and decide where to publish these ISOs and how to ensure they don't pile up. Pretty sure that currently most downloads are performed by search engine crawlers, not end users. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

Build failed in Jenkins: system-sync_mirrors-glusterfs-3.10-el7-x86_64 #1590
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-3.10-el7-x86_64/…> Changes: [Galit Rosenthal] Remove from lago fc27-EOL ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-3.10-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision fe8d4c3fee68d33b3407bba8a1408937639442a4 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f fe8d4c3fee68d33b3407bba8a1408937639442a4 Commit message: "Remove from lago fc27-EOL" > git rev-list --no-walk bb346506183a00e82791b798f0fbc591cc22957a # timeout=10 [system-sync_mirrors-glusterfs-3.10-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5426108617155492741.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror glusterfs-3.10-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1039, in _getFile raise e yum.Errors.NoMoreMirrorsRepoError: failure: repodata/repomd.xml from glusterfs-3.10-el7: [Errno 256] No more mirrors to try. http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.10/repodata/repo…: [Errno 14] HTTP Error 404 - Not Found Build step 'Execute shell' marked build as failure

1 2

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #1971
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by user Ehud Yonasi [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse 05b40dfb4ec43a82530e8c471395d1858e5c59e1^{commit} # timeout=10 > git rev-parse origin/05b40dfb4ec43a82530e8c471395d1858e5c59e1^{commit} # timeout=10 > git rev-parse 05b40dfb4ec43a82530e8c471395d1858e5c59e1^{commit} # timeout=10 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.

1 4

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Eyal Edri (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2586: ---------------------------------- [~yturgema(a)redhat.com] can we change the OST job not to read from the jenkins the artifact? It might overload the server which is already overloaded. can you read it from another location maybe? perhaps resources.ovirt.org? > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- Another strange result I see is that there has been around a 1000 downloads of the 4.2 node-ng ISOs from ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/exported-artifacts over the course of the last week. Not sure what's up with that but we may be advertising this Jenkins URL directly from the website and it should not be used for thousands of ISO downloads as these block UI worker threads for a long period of time and may become a contributing factor to UI responsiveness. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2613) add_host fails on master and 4.2 ovirt due to missing deps
by Dafna Ron (oVirt JIRA) 04 Dec '18

04 Dec '18

Dafna Ron created OVIRT-2613: -------------------------------- Summary: add_host fails on master and 4.2 ovirt due to missing deps Key: OVIRT-2613 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2613 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra Priority: Highest Gal Ben Haim 9:28 AM (3 hours ago) to rhv-devops, RHV-CI We are going to rebase [1] which will solve the deps error. We need to replace the image otherwise add_host fails with a segmentation fault. An image from virt-builder probably won't be available today, so we will build our own el7.6 image. [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11873/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- Had an issue now reported by [~eyonasi(a)redhat.com] that Jenkins was not responding. I was monitoring CPU and it was just 100% used (400-600% used normally and 800% during high load). So Jenkins was idling for a few minutes and logs seem to show long running requests finishing before it came back to normal: [04/Dec/2018:12:05:54 +0000] "POST /ajaxBuildQueue HTTP/1.1" 502 408 [04/Dec/2018:12:04:59 +0000] "GET /view/oVirt%20system%20tests/job/ovirt-system-tests_manual/build?delay=0sec HTTP/1.1" 502 497 [04/Dec/2018:12:05:45 +0000] "POST /ajaxExecutors HTTP/1.1" 502 406 [04/Dec/2018:12:05:49 +0000] "POST /ajaxExecutors HTTP/1.1" 502 406 [04/Dec/2018:12:05:03 +0000] "GET /job/system-sync_mirrors-centos-kvm-common-el7-x86_64/ HTTP/1.1" 502 485 [04/Dec/2018:12:05:44 +0000] "GET / HTTP/1.1" 502 379 [04/Dec/2018:12:05:44 +0000] "POST /ajaxBuildQueue HTTP/1.1" 502 408 [04/Dec/2018:12:03:47 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543923704114-zly0n HTTP/1.1" 200 139 [04/Dec/2018:12:04:07 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543923241074-hu9gfi HTTP/1.1" 200 112 [04/Dec/2018:12:04:07 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543917181943-ulooc HTTP/1.1" 200 111 [04/Dec/2018:12:03:40 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543915343510-2uggp HTTP/1.1" 200 112 [04/Dec/2018:12:04:08 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543844460885-fshsx HTTP/1.1" 200 112 [04/Dec/2018:12:03:45 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543916086881-v6zmif HTTP/1.1" 200 113 [04/Dec/2018:12:06:06 +0000] "GET /favicon.ico HTTP/1.1" 200 17542 As can be seen from this log, sse-gateway requests started around 12:03 terminated only after 12:06 which is just too long. As noted before, these requests are initiated from BlueOcean as it seems. [~bkorren(a)redhat.com] did we by chance recently add BlueOcean links to any report emails or something else that could increase popularity of BlueOcean and induce this behavior? There were no updates in the past month that could directly influence this so I'm trying to locate a possible root cause. I can note as well that there's a bunch of requests related to OpenShift yet I do not have any proof they are slowing doen the Master at this point. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #1968
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision fe8d4c3fee68d33b3407bba8a1408937639442a4 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f fe8d4c3fee68d33b3407bba8a1408937639442a4 Commit message: "Remove from lago fc27-EOL" > git rev-list --no-walk fe8d4c3fee68d33b3407bba8a1408937639442a4 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1601595848307799396.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1039, in _getFile raise e yum.Errors.NoMoreMirrorsRepoError: failure: repodata/repomd.xml from glusterfs-3.10-el7: [Errno 256] No more mirrors to try. http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.10/repodata/repo…: [Errno 14] HTTP Error 404 - Not Found Build step 'Execute shell' marked build as failure

1 2

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Failed to run project_setup.sh for: #2762 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-qhvsl.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Failed to run project_setup.sh for: #2761 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-r441f.

1 0

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-el7-x86_64 - Build # 133 - Failure!
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el7… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el7… Build Number: 133 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #133 [Yuval Turgeman] stdci: move build-artifacts to a v1 job [Evgheni Dereveanchin] Add distribution-gpg-keys to global_setup ----------------- Failed Tests: ----------------- No tests ran.

1 1

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Failed to run project_setup.sh for: #2760 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-rc8g7.

1 0

[CQ]: 95927,3 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 04 Dec '18

04 Dec '18

Change 95927,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/95927/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11873/

1 0

[oVirt Jenkins] ovirt-appliance_master_build-artifacts-fc28-x86_64 - Build # 142 - Failure!
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Project: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build Number: 142 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #142 [Evgheni Dereveanchin] Add distribution-gpg-keys to global_setup [Yuval Turgeman] automation: remove build-artifacts* from stdci-v2 ----------------- Failed Tests: ----------------- No tests ran.

1 1

[CQ]: 69193, 8 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69193,8 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69193/8 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11871/

1 0

[CQ]: 69069, 10 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69069,10 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69069/10 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11870/

1 0

[CQ]: 69190, 9 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69190,9 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69190/9 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11868/

1 0

[CQ]: 69193, 8 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69193,8 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69193/8 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11866/

1 0

[CQ]: 69069, 10 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69069,10 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69069/10 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11863/

1 0

[CQ]: 69190, 9 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69190,9 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69190/9 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11860/

1 0

[CQ]: 69189, 9 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69189,9 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69189/9 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11857/

1 0

[CQ]: 69189, 9 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 04 Dec '18

04 Dec '18

A system test invoked by the "ovirt-master" change queue including change 69189,9 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/69189/9 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11854/

1 0

oVirt infra daily report - unstable production jobs - 663
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/663//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 95918,7 (ovirt-imageio) failed "ovirt-master" system tests
by oVirt Jenkins 03 Dec '18

03 Dec '18

Change 95918,7 (ovirt-imageio) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/95918/7 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11851/

1 0

[CQ]: 252c21d (ovirt-ansible-engine-setup) failed "ovirt-master" system tests
by oVirt Jenkins 03 Dec '18

03 Dec '18

Change 252c21d (ovirt-ansible-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-engine-setup/commit/252c21d708e0e9a1… For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11850/

1 0

[CQ]: f81f7ce (ovirt-ansible) failed "ovirt-master" system tests
by oVirt Jenkins 03 Dec '18

03 Dec '18

Change f81f7ce (ovirt-ansible) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible/commit/f81f7ce0eb7ec1c8babc6afd49ed2… For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11848/

1 0

[CQ]: a1e4392 (ovirt-ansible) failed "ovirt-4.2" system tests
by oVirt Jenkins 03 Dec '18

03 Dec '18

Change a1e4392 (ovirt-ansible) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible/commit/a1e439270f84e4ee422996fc5b5df… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3582/

1 0

[CQ]: 95746, 11 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 03 Dec '18

03 Dec '18

A system test invoked by the "ovirt-master" change queue including change 95746,11 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95746/11 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11846/

1 0

[CQ]: 95746, 11 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 03 Dec '18

03 Dec '18

A system test invoked by the "ovirt-master" change queue including change 95746,11 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95911,1 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95911,1 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95746/11 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95911/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11843/

1 0

[CQ]: 252c21d (ovirt-ansible-engine-setup) failed "ovirt-4.2" system tests
by oVirt Jenkins 03 Dec '18

03 Dec '18

Change 252c21d (ovirt-ansible-engine-setup) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://github.com/oVirt/ovirt-ansible-engine-setup/commit/252c21d708e0e9a1… For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3581/

1 0

[JIRA] (OVIRT-2609) stdci style reports sometimes disappear pretty fast
by Barak Korren (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2609?page=com.atlassian.jira.… ] Barak Korren reassigned OVIRT-2609: ----------------------------------- Assignee: Ehud Yonasi (was: infra) > stdci style reports sometimes disappear pretty fast > --------------------------------------------------- > > Key: OVIRT-2609 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2609 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: Ehud Yonasi > > Can there be a SLA defined for how long we can see test results until they > get removed? > It does not have to be like on travis (where I can still see all my build > logs from years ago), but at least a predictable period would be nice. The > PR in [1] did for instance not have the results anymore from 5 days ago. > Thanks, > Roman > [1] https://github.com/kubevirt/kubevirt/pull/1744 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2609) stdci style reports sometimes disappear pretty fast
by Roman Mohr (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2609?page=com.atlassian.jira.… ] Roman Mohr commented on OVIRT-2609: ----------------------------------- [~bkorren(a)redhat.com] you see them because I triggered them today again. > stdci style reports sometimes disappear pretty fast > --------------------------------------------------- > > Key: OVIRT-2609 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2609 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: infra > > Can there be a SLA defined for how long we can see test results until they > get removed? > It does not have to be like on travis (where I can still see all my build > logs from years ago), but at least a predictable period would be nice. The > PR in [1] did for instance not have the results anymore from 5 days ago. > Thanks, > Roman > [1] https://github.com/kubevirt/kubevirt/pull/1744 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2609) stdci style reports sometimes disappear pretty fast
by Barak Korren (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2609?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2609: ------------------------------------- Its kept for 14 days, looking at the linked PR, I still see results there: https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/2735/arti… I do see we also have a setting to keep only the last 60 built artifacts, we may want to remove that. > stdci style reports sometimes disappear pretty fast > --------------------------------------------------- > > Key: OVIRT-2609 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2609 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: infra > > Can there be a SLA defined for how long we can see test results until they > get removed? > It does not have to be like on travis (where I can still see all my build > logs from years ago), but at least a predictable period would be nice. The > PR in [1] did for instance not have the results anymore from 5 days ago. > Thanks, > Roman > [1] https://github.com/kubevirt/kubevirt/pull/1744 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

Space issues on lfedora1.lf-dev.marist.edu
by Barak Korren 03 Dec '18

03 Dec '18

Hi Dan, How are you. As you know we've been using `lfedora1.lf-dev.marist.edu` to generate s390x build of oVirt. We've recently seen some failures that have to do with running our of space on the node. Some of this seems to be our fault, as clearing up stale mock chroots we created freed up about 14G, but after doing that I still see there are 48G used there (Is the OS image that big?). Can some more space be cleared up on the node? Could we perhaps have the disk space increased there? I was also wondering, could we setup and use Docker on that node? We've been switching to using containers on our regular CI nodes, and it'd be a shame to leave s390x behind... Thanks, Barak. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 5

[JIRA] (OVIRT-2155) remove Jenkins Jabber plugin
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2155?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-2155: ---------------------------------------- Issue Type: Sub-task (was: Task) Parent: OVIRT-2612 > remove Jenkins Jabber plugin > ---------------------------- > > Key: OVIRT-2155 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2155 > Project: oVirt - virtualization made easy > Issue Type: Sub-task > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Low > > We have the jabber-plugin installed (together with instant-messaging it depends on) that does not seem to be used nor do we have any plans to utilize this plugin in the future. > Opening ticket to remove the plugin from the puppet manifest and remove it manually from the Master -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2157) remove Jenkins Copr plugin
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2157?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-2157: ---------------------------------------- Issue Type: Sub-task (was: Task) Parent: OVIRT-2612 > remove Jenkins Copr plugin > -------------------------- > > Key: OVIRT-2157 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2157 > Project: oVirt - virtualization made easy > Issue Type: Sub-task > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Low > > We have the Copr plugin installed on Jenkins and not used by any jobs. > Opening ticket to drop the plugin from the puppet manifest and remove it manually from the Master -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2612) tickets to improve jenkins performance
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

Evgheni Dereveanchin created OVIRT-2612: ------------------------------------------- Summary: tickets to improve jenkins performance Key: OVIRT-2612 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2612 Project: oVirt - virtualization made easy Issue Type: Epic Reporter: Evgheni Dereveanchin Assignee: infra -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

Change in barak-test[master]: jenkins integ test
by review＠gerrit-staging.phx.ovirt.org 03 Dec '18

03 Dec '18

From Evgheni Dereveanchin <ederevea(a)redhat.com>: Evgheni Dereveanchin has posted comments on this change. ( https://gerrit-staging.phx.ovirt.org/17 ) Change subject: jenkins integ test ...................................................................... Patch Set 1: OVIRT-2606 test -- To view, visit https://gerrit-staging.phx.ovirt.org/17 To unsubscribe, visit https://gerrit-staging.phx.ovirt.org/settings Gerrit-Project: barak-test Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50acc69c8c5b6b3e03f3ced23d72c883ac4db2c2 Gerrit-Change-Number: 17 Gerrit-PatchSet: 1 Gerrit-Owner: Evgheni Dereveanchin <ederevea(a)redhat.com> Gerrit-Reviewer: Evgheni Dereveanchin <ederevea(a)redhat.com> Gerrit-Reviewer: Jenkins CI <infra(a)ovirt.org> Gerrit-Comment-Date: Mon, 03 Dec 2018 16:54:01 +0000 Gerrit-HasComments: No

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-2586: ---------------------------------------- Issue Type: Outage (was: By-EMAIL) > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- Here's a memory histogram for top consumers: ||Class||Size (Kb)||% size||Instances||% instances|| |org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$Retry| 4,259,284| 32 |136,297,090 |32 | |java.util.concurrent.ConcurrentLinkedQueue$Node| 3,194,540| 24 |136,300,398 |32 | |java.lang.Object[]| 1,389,218| 10 |25,260,454 |5 | |java.util.ArrayList| 568,681| 4 |24,263,747 |5 | |hudson.plugins.git.Revision| 557,909| 4 |23,804,146 |5 | |java.util.HashMap$Node| 406,807| 3 |13,017,830 |3 | |hudson.plugins.git.util.Build| 372,095| 2 |11,907,066 |2 | |org.eclipse.jgit.lib.ObjectId| 364,458| 2 |11,662,676 |2 | |java.lang.StackTraceElement| 280,683| 2 |8,981,873 |2 | |hudson.plugins.git.Branch| 273,343| 2 |11,662,674 |2 | |char[]| 228,907| 1 |1,744,589 |0 | |byte[]| 145,645| 1 |60,719 |0 | |java.lang.String| 141,084| 1 |6,019,605 |1 | ssegateway is way at the top and it's used by the BlueOcean events API This is not only affecting the UI as it seems but also the tasks that go through the API so we have issues such as OVIRT-2608 for example. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JENKINS] Failed to setup proejct kubevirt_containerized-data-importer_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 03 Dec '18

03 Dec '18

Failed to run project_setup.sh for: #839 containerized-data-importer [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-9x945.

1 0

[JIRA] (OVIRT-2611) Move "ovirt-srv17" to u/s openshift for testing purposes
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2611?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2611: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Move "ovirt-srv17" to u/s openshift for testing purposes > -------------------------------------------------------- > > Key: OVIRT-2611 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2611 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Gal Ben Haim > Assignee: Evgheni Dereveanchin > > Hi, > Please move "ovirt-srv17" from Jenkins to u/s Openshift. For the meanwhile don't add to it the `type: bare-metal` label, since I need it for performance investigation (we want to understand why OST jobs take longer on Openshift). > cc [~dbelenky(a)redhat.com] [~bkorren(a)redhat.com] -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2611) Move "ovirt-srv17" to u/s openshift for testing purposes
by Gal Ben Haim (oVirt JIRA) 03 Dec '18

03 Dec '18

Gal Ben Haim created OVIRT-2611: ----------------------------------- Summary: Move "ovirt-srv17" to u/s openshift for testing purposes Key: OVIRT-2611 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2611 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Gal Ben Haim Assignee: infra Hi, Please move "ovirt-srv17" from Jenkins to u/s Openshift. For the meanwhile don't add to it the `type: bare-metal` label, since I need it for performance investigation (we want to understand why OST jobs take longer on Openshift). cc [~dbelenky(a)redhat.com] [~bkorren(a)redhat.com] -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2609) stdci style reports sometimes disappear pretty fast
by Eyal Edri (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2609?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2609: ---------------------------------- [~ederevea] what is the current retention policy for builds history? How much can we extend it without affecting performance and space too much? > stdci style reports sometimes disappear pretty fast > --------------------------------------------------- > > Key: OVIRT-2609 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2609 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: infra > > Can there be a SLA defined for how long we can see test results until they > get removed? > It does not have to be like on travis (where I can still see all my build > logs from years ago), but at least a predictable period would be nice. The > PR in [1] did for instance not have the results anymore from 5 days ago. > Thanks, > Roman > [1] https://github.com/kubevirt/kubevirt/pull/1744 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2610) Add alerts for Jenkins job queue
by Evgheni Dereveanchin (oVirt JIRA) 03 Dec '18

03 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2610?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2610: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Add alerts for Jenkins job queue > -------------------------------- > > Key: OVIRT-2610 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2610 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > We do have some monitoring in place for Jenkins job queues yet there are no alerts configured. > Opening ticket to discuss adding alerts and normal actions that should be taken if one happens. > Sample actions for the infra owner: > * verify that jobs aren't stuck waiting for a label that has no hosts assigned > * verify that a stuck job isn't blocking queues (CQ in particular) > Monitoring can be added to Nagios using a trivial plugin based on API output: > # curl jenkins.ovirt.org/queue/api/xml | grep -o '<item' | wc -l -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0