December 2018 - Infra - oVirt List Archives

Jenkins build became unstable: ovirt_master_publish-rpms_nightly #1201
by jenkins＠jenkins.phx.ovirt.org 26 Dec '18

26 Dec '18

See <http://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/1201/display…>

1 4

Tests failed because global_setup.sh failed
by Nir Soffer 26 Dec '18

26 Dec '18

Not sure why global setup failed: + [[ ! -O /home/jenkins/.ssh ]] + [[ ! -G /home/jenkins/.ssh ]] + verify_set_permissions 700 /home/jenkins/.ssh + local target_permissions=700 + local path_to_set=/home/jenkins/.ssh ++ stat -c %a /home/jenkins/.ssh + local access=700 + [[ 700 != \7\0\0 ]] + return 0 + [[ -f /home/jenkins/.ssh/known_hosts ]] + verify_set_ownership /home/jenkins/.ssh/known_hosts + local path_to_set=/home/jenkins/.ssh/known_hosts ++ id -un + local owner=jenkins ++ id -gn + local group=jenkins + [[ ! -O /home/jenkins/.ssh/known_hosts ]] + [[ ! -G /home/jenkins/.ssh/known_hosts ]] + verify_set_permissions 644 /home/jenkins/.ssh/known_hosts + local target_permissions=644 + local path_to_set=/home/jenkins/.ssh/known_hosts ++ stat -c %a /home/jenkins/.ssh/known_hosts + local access=644 + [[ 644 != \6\4\4 ]] + return 0 + return 0 + true + log ERROR Aborting. Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st…

4 4

oVirt infra daily report - unstable production jobs - 685
by jenkins＠jenkins.phx.ovirt.org 26 Dec '18

26 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/685//artifact/exported-a… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2538) ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2538?page=com.atlassian.jira.… ] Eyal Edri updated OVIRT-2538: ----------------------------- Resolution: Cannot Reproduce Status: Done (was: To Do) > ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome > --------------------------------------------------------------------------- > > Key: OVIRT-2538 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2538 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: High > Labels: ost_failures, ost_race > > I have seen random failures of test 008_basic_ui_sanity.initialize_chrome which are not related to the changes. > Here is the latest failure which I confirmed with didi is not a real failure and is not related to the change > https://gerrit.ovirt.org/#/c/89572/ > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10601/ > I cannot see anything in the http logs and in engine logs. > I can see a warn on ssl and some issues which seems to be resolved later on domain names: > '[Mon Oct 15 02:56:26.041204 2018] [ssl:warn] [pid 18661] AH01909: RSA certificate configured for 192.168.201.4:443 does NOT include an ID which matches the server name' > I think the issue is the test since the only errors I can see are in the lago logs. > I saved logs to allow debugging of the test failure. > [~gbenhaim(a)redhat.com] > [~gshereme(a)redhat.com] > Can you please check this issue? -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2538) ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2538?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2538: ---------------------------------- I’m not sure its worth investing time if it fails 0.05% of the time. AFAIK test is passing for now and we don’t have a pending issue, so I’ll close this one for now. If you have any ideas/suggestions to improve the UI test, feel free to submit a patch to OST \:) > ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome > --------------------------------------------------------------------------- > > Key: OVIRT-2538 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2538 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: High > Labels: ost_failures, ost_race > > I have seen random failures of test 008_basic_ui_sanity.initialize_chrome which are not related to the changes. > Here is the latest failure which I confirmed with didi is not a real failure and is not related to the change > https://gerrit.ovirt.org/#/c/89572/ > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10601/ > I cannot see anything in the http logs and in engine logs. > I can see a warn on ssl and some issues which seems to be resolved later on domain names: > '[Mon Oct 15 02:56:26.041204 2018] [ssl:warn] [pid 18661] AH01909: RSA certificate configured for 192.168.201.4:443 does NOT include an ID which matches the server name' > I think the issue is the test since the only errors I can see are in the lago logs. > I saved logs to allow debugging of the test failure. > [~gbenhaim(a)redhat.com] > [~gshereme(a)redhat.com] > Can you please check this issue? -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-1856) Document procedure for infra upgrades
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1856?page=com.atlassian.jira.… ] Eyal Edri updated OVIRT-1856: ----------------------------- Resolution: Fixed Status: Done (was: To Do) We've added some steps to the ovirt-infra docs about upgrades, and it also continues to change as we upgrade the infra. so closing for now. > Document procedure for infra upgrades > -------------------------------------- > > Key: OVIRT-1856 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1856 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Eyal Edri > Assignee: Evgheni Dereveanchin > Priority: High > > On Sun, Jan 21, 2018 at 1:01 PM, Barak Korren <bkorren(a)redhat.com> wrote: > > > > > > On 21 January 2018 at 12:50, Eyal Edri <eedri(a)redhat.com> wrote: > > > >> > >> > >> On Sun, Jan 21, 2018 at 12:47 PM, Barak Korren <bkorren(a)redhat.com> > >> wrote: > >> > >>> > >>> > >>> On 21 January 2018 at 12:39, Eyal Edri <eedri(a)redhat.com> wrote: > >>> > >>>> There is another issue, which is currently failing all CQ, and its > >>>> related to the new IBRS CPU model. > >>>> It looks like all of the lago slaves were upgraded to new Libvirt and > >>>> kernel on Friday, while we still don't have a fix on lago-ost-plugin for > >>>> that. > >>>> > >>>> I think there was a misunderstanding about what to upgrade, and it > >>>> might have been understood that only the bios upgrade breaks it and not the > >>>> kernel one. > >>>> > >>>> In any case, we're currently fixing the issue, either by downgrading > >>>> the relevant pkgs on lago slaves or adding the mapping to new CPU types > >>>> from OST. > >>>> > >>>> For future, I suggest a few updates to maintenance work on Jenkins > >>>> slaves ( VMs or BM ): > >>>> > >>>> 1. Let's avoid doing an upgrade close to a weekend ( i.e not on Thu-Sun > >>>> ), so all the team can be around to help if needed or if something > >>>> unexpected happens. > >>>> 2. When we have a system-wide upgrade scheduled, like all BM slaves or > >>>> VMs for a specific OS, let's adopt a gradual upgrade with a few days window > >>>> in between, > >>>> e.g, if we need to upgrade all Lago slaves, let's upgrade 1-2 and > >>>> wait to see if nothing breaks and continue after we verify OST runs ( > >>>> either seeing on CQ or running manually ) > >>>> > >>>> > >>>> Thoughts? > >>>> > >>>> > >>> We have a staging system - we should be using it for staging.... > >>> > >> > >> Do we have OST tests or manual job avaialble there? > >> > > > > We can add them easily, or simply run Lago manually when needed. > > > > > >> In any case, this doesn't contradict what I suggested, even if you test > >> on staging, there could be differences from the production system, so we > >> should take care when we upgrade regardless. > >> > > > > Yes, but at least we'd know we green lighted the new configuration - I'm > > sure in this case we could have found at least some of the issues on > > staging (Like the fc27 issues for example) and could have avoided expansive > > production failures. > > > > Another point when scheduling an upgrade, is to talk to infra owner or the > >> CI team and understand if we currently have a large Q in CQ or known > >> failures, so it might be best to wait a bit until its cleared. > >> > >> > > > > > Adding infra-support so we can gather this info and prepare a maintanaince > / upgrade checklist to add to the oVirt infra docs. > Let's continue the discussion, suggestion on that ticket. > > -- > > Barak Korren > > RHV DevOps team , RHCE, RHCi > > Red Hat EMEA > > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > > > -- > Eyal edri > MANAGER > RHV DevOps > EMEA VIRTUALIZATION R&D > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2631) [VDSM] Build fail with "Curl error (28): Timeout was reached for..."
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2631?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2631: ---------------------------------- [~ederevea] let's see after the shutdown if we can see any unusual slow network, or problem with our PHX network. > [VDSM] Build fail with "Curl error (28): Timeout was reached for..." > -------------------------------------------------------------------- > > Key: OVIRT-2631 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2631 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: Evgheni Dereveanchin > > Looks like creating mock env fail because of slow network. > Can we have local mirror to avoid such issues? > Error: > # /usr/bin/dnf --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/ > --releasever 28 --disableplugin=local --setopt=deltarpm=False install > @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git > iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw > make mom openvswitch ovirt-imageio-common policycoreutils-python > python2-augeas python2-blivet1 python2-coverage python2-dateutil > python2-dbus python2-decorator python2-devel python2-dmidecode > python2-enum34 python2-inotify python2-ioprocess python2-ipaddress > python2-magic python2-mock python2-netaddr python2-pthreading > python2-pyudev python2-pyyaml python2-requests python2-setuptools > python2-six python2-subprocess32 python3-PyYAML python3-augeas > python3-coverage python3-dateutil python3-dbus python3-decorator > python3-devel python3-magic python3-netaddr python3-nose > python3-pyudev python3-six python3-yaml rpm-build > ... > Transaction Summary > ================================================================================ > Install 675 Packages > Total download size: 424 M > Installed size: 1.5 G > Downloading Packages: > ... > (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00 > (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00 > (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00 > [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was > reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was > reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > Error: Error downloading packages: > Curl error (28): Timeout was reached for > http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2446) Github push trigger will fail if was trigger by a tag push
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2446?page=com.atlassian.jira.… ] Eyal Edri updated OVIRT-2446: ----------------------------- Blocked By: blocked until we implement tag stage for STD-CI Status: Blocked (was: To Do) [~eyonasi@redhat.com][~bkorren@redhat.com] please add links to the tag stage tickets. > Github push trigger will fail if was trigger by a tag push > ---------------------------------------------------------- > > Key: OVIRT-2446 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2446 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roy Golan > Assignee: infra > > When pushing a tag to a GH it triggers the jenkins webhook. In the invoked > job, > it will try to fech a new head with that tag, but that will fail since the > repo doesn't have > a head since it is started as git init. > A breakdown from > https://jenkins.ovirt.org/job/oVirt_ovirt-openshift-extensions_standard-on-… > - git init a repo > *14:57:15* > git init > /home/jenkins/workspace/oVirt_ovirt-openshift-extensions_standard-on-ghpush/ovirt-openshift-extensions > # timeout=10 > - git fetch tags (maybe the problem is here as it doesn't reffer to refs/tags) > *15:35:07* [check-merged.el7.x86_64] > git fetch --tags --progress > https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/* > - fetch upstream changes, which is here a tag v0.3.2, so it uses > 'refs/tags/v0.3.2' > *14:57:18* > git fetch --tags --progress > https://github.com/oVirt/ovirt-openshift-extensions > +refs/tags/v0.3.2:myhead > This fails with > *14:57:19* stderr: error: cannot update ref 'refs/heads/myhead': > trying to write non-commit object > 3c7b5a5d243d1f3bcecbe7b527726b06113d0ec5 to branch 'refs/heads/myhead' > While in regular merge with a tag on it uses 'refs/heads/master': > *15:35:10* [check-merged.el7.x86_64] > git fetch --tags --progress > https://gerrit.ovirt.org/jenkins +refs/heads/master:myhead > Since refs/tags/v0.3.2 doesn't exist locally it can't reference it. Since > refs/heads/master exists locally it can create the alias 'myhead' to it. > I would like to have the option to push a tag and trigger a build but this > is not working atm without a commit attached to it. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2446) Github push trigger will fail if was trigger by a tag push
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2446?page=com.atlassian.jira.… ] Eyal Edri updated OVIRT-2446: ----------------------------- Issue Type: New Feature (was: By-EMAIL) > Github push trigger will fail if was trigger by a tag push > ---------------------------------------------------------- > > Key: OVIRT-2446 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2446 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roy Golan > Assignee: infra > > When pushing a tag to a GH it triggers the jenkins webhook. In the invoked > job, > it will try to fech a new head with that tag, but that will fail since the > repo doesn't have > a head since it is started as git init. > A breakdown from > https://jenkins.ovirt.org/job/oVirt_ovirt-openshift-extensions_standard-on-… > - git init a repo > *14:57:15* > git init > /home/jenkins/workspace/oVirt_ovirt-openshift-extensions_standard-on-ghpush/ovirt-openshift-extensions > # timeout=10 > - git fetch tags (maybe the problem is here as it doesn't reffer to refs/tags) > *15:35:07* [check-merged.el7.x86_64] > git fetch --tags --progress > https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/* > - fetch upstream changes, which is here a tag v0.3.2, so it uses > 'refs/tags/v0.3.2' > *14:57:18* > git fetch --tags --progress > https://github.com/oVirt/ovirt-openshift-extensions > +refs/tags/v0.3.2:myhead > This fails with > *14:57:19* stderr: error: cannot update ref 'refs/heads/myhead': > trying to write non-commit object > 3c7b5a5d243d1f3bcecbe7b527726b06113d0ec5 to branch 'refs/heads/myhead' > While in regular merge with a tag on it uses 'refs/heads/master': > *15:35:10* [check-merged.el7.x86_64] > git fetch --tags --progress > https://gerrit.ovirt.org/jenkins +refs/heads/master:myhead > Since refs/tags/v0.3.2 doesn't exist locally it can't reference it. Since > refs/heads/master exists locally it can create the alias 'myhead' to it. > I would like to have the option to push a tag and trigger a build but this > is not working atm without a commit attached to it. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2533) multi-stage support in CI slaves
by Eyal Edri (oVirt JIRA) 25 Dec '18

25 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2533?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2533: ---------------------------------- i still don't know if we can support it, still needs to be discussed and scoped. [~bkorren(a)redhat.com] feel free to update and add info. > multi-stage support in CI slaves > -------------------------------- > > Key: OVIRT-2533 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2533 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Roy Golan > Assignee: infra > > I would like to get multi-stage support upstream. Our default docker > packages for centos 7.5 is 1.13. For multi-stage support we need > 1.17 or > have buildah working in mock (preferable over docker, won't happen now I > guess?). > Using this method[1] to install docker-ce I was able to get 1.18 running. > Doable? > Regards, > Roy > [1] https://docs.docker.com/install/linux/docker-ce/centos/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0