Tests failed because global_setup.sh failed
by Nir Soffer
Not sure why global setup failed:
+ [[ ! -O /home/jenkins/.ssh ]]
+ [[ ! -G /home/jenkins/.ssh ]]
+ verify_set_permissions 700 /home/jenkins/.ssh
+ local target_permissions=700
+ local path_to_set=/home/jenkins/.ssh
++ stat -c %a /home/jenkins/.ssh
+ local access=700
+ [[ 700 != \7\0\0 ]]
+ return 0
+ [[ -f /home/jenkins/.ssh/known_hosts ]]
+ verify_set_ownership /home/jenkins/.ssh/known_hosts
+ local path_to_set=/home/jenkins/.ssh/known_hosts
++ id -un
+ local owner=jenkins
++ id -gn
+ local group=jenkins
+ [[ ! -O /home/jenkins/.ssh/known_hosts ]]
+ [[ ! -G /home/jenkins/.ssh/known_hosts ]]
+ verify_set_permissions 644 /home/jenkins/.ssh/known_hosts
+ local target_permissions=644
+ local path_to_set=/home/jenkins/.ssh/known_hosts
++ stat -c %a /home/jenkins/.ssh/known_hosts
+ local access=644
+ [[ 644 != \6\4\4 ]]
+ return 0
+ return 0
+ true
+ log ERROR Aborting.
Build:
https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
5 years, 11 months
[JIRA] (OVIRT-2538) ovirt-engine master failure on test
008_basic_ui_sanity.initialize_chrome
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2538?page=com.atlassian.jir... ]
Eyal Edri updated OVIRT-2538:
-----------------------------
Resolution: Cannot Reproduce
Status: Done (was: To Do)
> ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome
> ---------------------------------------------------------------------------
>
> Key: OVIRT-2538
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2538
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: infra
> Priority: High
> Labels: ost_failures, ost_race
>
> I have seen random failures of test 008_basic_ui_sanity.initialize_chrome which are not related to the changes.
> Here is the latest failure which I confirmed with didi is not a real failure and is not related to the change
> https://gerrit.ovirt.org/#/c/89572/
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10601/
> I cannot see anything in the http logs and in engine logs.
> I can see a warn on ssl and some issues which seems to be resolved later on domain names:
> '[Mon Oct 15 02:56:26.041204 2018] [ssl:warn] [pid 18661] AH01909: RSA certificate configured for 192.168.201.4:443 does NOT include an ID which matches the server name'
> I think the issue is the test since the only errors I can see are in the lago logs.
> I saved logs to allow debugging of the test failure.
> [~gbenhaim(a)redhat.com]
> [~gshereme(a)redhat.com]
> Can you please check this issue?
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-2538) ovirt-engine master failure on test
008_basic_ui_sanity.initialize_chrome
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2538?page=com.atlassian.jir... ]
Eyal Edri commented on OVIRT-2538:
----------------------------------
I’m not sure its worth investing time if it fails 0.05% of the time.
AFAIK test is passing for now and we don’t have a pending issue, so I’ll close this one for now.
If you have any ideas/suggestions to improve the UI test, feel free to submit a patch to OST \:)
> ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome
> ---------------------------------------------------------------------------
>
> Key: OVIRT-2538
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2538
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: infra
> Priority: High
> Labels: ost_failures, ost_race
>
> I have seen random failures of test 008_basic_ui_sanity.initialize_chrome which are not related to the changes.
> Here is the latest failure which I confirmed with didi is not a real failure and is not related to the change
> https://gerrit.ovirt.org/#/c/89572/
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10601/
> I cannot see anything in the http logs and in engine logs.
> I can see a warn on ssl and some issues which seems to be resolved later on domain names:
> '[Mon Oct 15 02:56:26.041204 2018] [ssl:warn] [pid 18661] AH01909: RSA certificate configured for 192.168.201.4:443 does NOT include an ID which matches the server name'
> I think the issue is the test since the only errors I can see are in the lago logs.
> I saved logs to allow debugging of the test failure.
> [~gbenhaim(a)redhat.com]
> [~gshereme(a)redhat.com]
> Can you please check this issue?
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-1856) Document procedure for infra upgrades
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1856?page=com.atlassian.jir... ]
Eyal Edri updated OVIRT-1856:
-----------------------------
Resolution: Fixed
Status: Done (was: To Do)
We've added some steps to the ovirt-infra docs about upgrades, and it also continues to change as we upgrade the infra. so closing for now.
> Document procedure for infra upgrades
> --------------------------------------
>
> Key: OVIRT-1856
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1856
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Eyal Edri
> Assignee: Evgheni Dereveanchin
> Priority: High
>
> On Sun, Jan 21, 2018 at 1:01 PM, Barak Korren <bkorren(a)redhat.com> wrote:
> >
> >
> > On 21 January 2018 at 12:50, Eyal Edri <eedri(a)redhat.com> wrote:
> >
> >>
> >>
> >> On Sun, Jan 21, 2018 at 12:47 PM, Barak Korren <bkorren(a)redhat.com>
> >> wrote:
> >>
> >>>
> >>>
> >>> On 21 January 2018 at 12:39, Eyal Edri <eedri(a)redhat.com> wrote:
> >>>
> >>>> There is another issue, which is currently failing all CQ, and its
> >>>> related to the new IBRS CPU model.
> >>>> It looks like all of the lago slaves were upgraded to new Libvirt and
> >>>> kernel on Friday, while we still don't have a fix on lago-ost-plugin for
> >>>> that.
> >>>>
> >>>> I think there was a misunderstanding about what to upgrade, and it
> >>>> might have been understood that only the bios upgrade breaks it and not the
> >>>> kernel one.
> >>>>
> >>>> In any case, we're currently fixing the issue, either by downgrading
> >>>> the relevant pkgs on lago slaves or adding the mapping to new CPU types
> >>>> from OST.
> >>>>
> >>>> For future, I suggest a few updates to maintenance work on Jenkins
> >>>> slaves ( VMs or BM ):
> >>>>
> >>>> 1. Let's avoid doing an upgrade close to a weekend ( i.e not on Thu-Sun
> >>>> ), so all the team can be around to help if needed or if something
> >>>> unexpected happens.
> >>>> 2. When we have a system-wide upgrade scheduled, like all BM slaves or
> >>>> VMs for a specific OS, let's adopt a gradual upgrade with a few days window
> >>>> in between,
> >>>> e.g, if we need to upgrade all Lago slaves, let's upgrade 1-2 and
> >>>> wait to see if nothing breaks and continue after we verify OST runs (
> >>>> either seeing on CQ or running manually )
> >>>>
> >>>>
> >>>> Thoughts?
> >>>>
> >>>>
> >>> We have a staging system - we should be using it for staging....
> >>>
> >>
> >> Do we have OST tests or manual job avaialble there?
> >>
> >
> > We can add them easily, or simply run Lago manually when needed.
> >
> >
> >> In any case, this doesn't contradict what I suggested, even if you test
> >> on staging, there could be differences from the production system, so we
> >> should take care when we upgrade regardless.
> >>
> >
> > Yes, but at least we'd know we green lighted the new configuration - I'm
> > sure in this case we could have found at least some of the issues on
> > staging (Like the fc27 issues for example) and could have avoided expansive
> > production failures.
> >
> > Another point when scheduling an upgrade, is to talk to infra owner or the
> >> CI team and understand if we currently have a large Q in CQ or known
> >> failures, so it might be best to wait a bit until its cleared.
> >>
> >>
> >
> >
> Adding infra-support so we can gather this info and prepare a maintanaince
> / upgrade checklist to add to the oVirt infra docs.
> Let's continue the discussion, suggestion on that ticket.
> > --
> > Barak Korren
> > RHV DevOps team , RHCE, RHCi
> > Red Hat EMEA
> > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
> >
> --
> Eyal edri
> MANAGER
> RHV DevOps
> EMEA VIRTUALIZATION R&D
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-2631) [VDSM] Build fail with "Curl error (28):
Timeout was reached for..."
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2631?page=com.atlassian.jir... ]
Eyal Edri commented on OVIRT-2631:
----------------------------------
[~ederevea] let's see after the shutdown if we can see any unusual slow network, or problem with our PHX network.
> [VDSM] Build fail with "Curl error (28): Timeout was reached for..."
> --------------------------------------------------------------------
>
> Key: OVIRT-2631
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2631
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: Evgheni Dereveanchin
>
> Looks like creating mock env fail because of slow network.
> Can we have local mirror to avoid such issues?
> Error:
> # /usr/bin/dnf --installroot
> /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/
> --releasever 28 --disableplugin=local --setopt=deltarpm=False install
> @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git
> iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw
> make mom openvswitch ovirt-imageio-common policycoreutils-python
> python2-augeas python2-blivet1 python2-coverage python2-dateutil
> python2-dbus python2-decorator python2-devel python2-dmidecode
> python2-enum34 python2-inotify python2-ioprocess python2-ipaddress
> python2-magic python2-mock python2-netaddr python2-pthreading
> python2-pyudev python2-pyyaml python2-requests python2-setuptools
> python2-six python2-subprocess32 python3-PyYAML python3-augeas
> python3-coverage python3-dateutil python3-dbus python3-decorator
> python3-devel python3-magic python3-netaddr python3-nose
> python3-pyudev python3-six python3-yaml rpm-build
> ...
> Transaction Summary
> ================================================================================
> Install 675 Packages
> Total download size: 424 M
> Installed size: 1.5 G
> Downloading Packages:
> ...
> (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00
> (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00
> (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00
> [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
> reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
> reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> Error: Error downloading packages:
> Curl error (28): Timeout was reached for
> http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> Build:
> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-2446) Github push trigger will fail if was trigger by
a tag push
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2446?page=com.atlassian.jir... ]
Eyal Edri updated OVIRT-2446:
-----------------------------
Blocked By: blocked until we implement tag stage for STD-CI
Status: Blocked (was: To Do)
[~eyonasi@redhat.com][~bkorren(a)redhat.com] please add links to the tag stage tickets.
> Github push trigger will fail if was trigger by a tag push
> ----------------------------------------------------------
>
> Key: OVIRT-2446
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2446
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Reporter: Roy Golan
> Assignee: infra
>
> When pushing a tag to a GH it triggers the jenkins webhook. In the invoked
> job,
> it will try to fech a new head with that tag, but that will fail since the
> repo doesn't have
> a head since it is started as git init.
> A breakdown from
> https://jenkins.ovirt.org/job/oVirt_ovirt-openshift-extensions_standard-o...
> - git init a repo
> *14:57:15* > git init
> /home/jenkins/workspace/oVirt_ovirt-openshift-extensions_standard-on-ghpush/ovirt-openshift-extensions
> # timeout=10
> - git fetch tags (maybe the problem is here as it doesn't reffer to refs/tags)
> *15:35:07* [check-merged.el7.x86_64] > git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*
> - fetch upstream changes, which is here a tag v0.3.2, so it uses
> 'refs/tags/v0.3.2'
> *14:57:18* > git fetch --tags --progress
> https://github.com/oVirt/ovirt-openshift-extensions
> +refs/tags/v0.3.2:myhead
> This fails with
> *14:57:19* stderr: error: cannot update ref 'refs/heads/myhead':
> trying to write non-commit object
> 3c7b5a5d243d1f3bcecbe7b527726b06113d0ec5 to branch 'refs/heads/myhead'
> While in regular merge with a tag on it uses 'refs/heads/master':
> *15:35:10* [check-merged.el7.x86_64] > git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/master:myhead
> Since refs/tags/v0.3.2 doesn't exist locally it can't reference it. Since
> refs/heads/master exists locally it can create the alias 'myhead' to it.
> I would like to have the option to push a tag and trigger a build but this
> is not working atm without a commit attached to it.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-2446) Github push trigger will fail if was trigger by
a tag push
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2446?page=com.atlassian.jir... ]
Eyal Edri updated OVIRT-2446:
-----------------------------
Issue Type: New Feature (was: By-EMAIL)
> Github push trigger will fail if was trigger by a tag push
> ----------------------------------------------------------
>
> Key: OVIRT-2446
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2446
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Reporter: Roy Golan
> Assignee: infra
>
> When pushing a tag to a GH it triggers the jenkins webhook. In the invoked
> job,
> it will try to fech a new head with that tag, but that will fail since the
> repo doesn't have
> a head since it is started as git init.
> A breakdown from
> https://jenkins.ovirt.org/job/oVirt_ovirt-openshift-extensions_standard-o...
> - git init a repo
> *14:57:15* > git init
> /home/jenkins/workspace/oVirt_ovirt-openshift-extensions_standard-on-ghpush/ovirt-openshift-extensions
> # timeout=10
> - git fetch tags (maybe the problem is here as it doesn't reffer to refs/tags)
> *15:35:07* [check-merged.el7.x86_64] > git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*
> - fetch upstream changes, which is here a tag v0.3.2, so it uses
> 'refs/tags/v0.3.2'
> *14:57:18* > git fetch --tags --progress
> https://github.com/oVirt/ovirt-openshift-extensions
> +refs/tags/v0.3.2:myhead
> This fails with
> *14:57:19* stderr: error: cannot update ref 'refs/heads/myhead':
> trying to write non-commit object
> 3c7b5a5d243d1f3bcecbe7b527726b06113d0ec5 to branch 'refs/heads/myhead'
> While in regular merge with a tag on it uses 'refs/heads/master':
> *15:35:10* [check-merged.el7.x86_64] > git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/master:myhead
> Since refs/tags/v0.3.2 doesn't exist locally it can't reference it. Since
> refs/heads/master exists locally it can create the alias 'myhead' to it.
> I would like to have the option to push a tag and trigger a build but this
> is not working atm without a commit attached to it.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
