Fwd: ** PROBLEM Service Alert: Resources file server/Resources
Partition is WARNING **
by Eyal Edri
let's see what we can do tomorrow to resolve this.
---------- Forwarded message ---------
From: <nagios(a)monitoring.ovirt.org>
Date: Sun, Dec 23, 2018, 17:19
Subject: ** PROBLEM Service Alert: Resources file server/Resources
Partition is WARNING **
To: <eedri(a)redhat.com>
***** Nagios *****
Notification Type: PROBLEM
Service: Resources Partition
Host: Resources file server
Address: resources.ovirt.org
State: WARNING
Date/Time: Sun Dec 23 15:19:00 UTC 2018
Additional Info:
DISK WARNING - free space: /srv/resources 167632 MB (14.57% inode=100%):
5 years, 11 months
[JIRA] (OVIRT-2631) [VDSM] Build fail with "Curl error (28):
Timeout was reached for..."
by Eyal Edri (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2631?page=com.atlassian.jir... ]
Eyal Edri commented on OVIRT-2631:
----------------------------------
Galit,
Can you check if this is still an issue?
On Thu, Dec 20, 2018 at 3:03 PM Eyal Edri <eedri(a)redhat.com> wrote:
> Evgheni,
> Do we have an issue with our proxy or its a problem in Fedora mirrors?
>
> On Thu, Dec 20, 2018 at 2:25 PM Nir Soffer <nsoffer(a)redhat.com> wrote:
>
>> Looks like creating mock env fail because of slow network.
>>
>> Can we have local mirror to avoid such issues?
>>
>> Error:
>>
>> # /usr/bin/dnf --installroot
>>
>> /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/
>> --releasever 28 --disableplugin=local --setopt=deltarpm=False install
>> @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git
>> iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw
>> make mom openvswitch ovirt-imageio-common policycoreutils-python
>> python2-augeas python2-blivet1 python2-coverage python2-dateutil
>> python2-dbus python2-decorator python2-devel python2-dmidecode
>> python2-enum34 python2-inotify python2-ioprocess python2-ipaddress
>> python2-magic python2-mock python2-netaddr python2-pthreading
>> python2-pyudev python2-pyyaml python2-requests python2-setuptools
>> python2-six python2-subprocess32 python3-PyYAML python3-augeas
>> python3-coverage python3-dateutil python3-dbus python3-decorator
>> python3-devel python3-magic python3-netaddr python3-nose
>> python3-pyudev python3-six python3-yaml rpm-build
>> ...
>> Transaction Summary
>>
>> ================================================================================
>> Install 675 Packages
>>
>> Total download size: 424 M
>> Installed size: 1.5 G
>> Downloading Packages:
>> ...
>> (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB
>> 00:00
>> (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB
>> 00:00
>> (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB
>> 00:00
>> [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
>> reached for
>> http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
>> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>> seconds]
>> [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
>> reached for
>> http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
>> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>> seconds]
>> Error: Error downloading packages:
>> Curl error (28): Timeout was reached for
>>
>> http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
>> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
>> seconds]
>>
>>
>> Build:
>>
>> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
>> _______________________________________________
>> Infra mailing list -- infra(a)ovirt.org
>> To unsubscribe send an email to infra-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/EMCHJTOHQYD...
>>
>
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV/CNV DevOps
>
> EMEA VIRTUALIZATION R&D
>
>
> Red Hat EMEA <https://www.redhat.com/>
> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
--
Eyal edri
MANAGER
RHV/CNV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
> [VDSM] Build fail with "Curl error (28): Timeout was reached for..."
> --------------------------------------------------------------------
>
> Key: OVIRT-2631
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2631
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: infra
>
> Looks like creating mock env fail because of slow network.
> Can we have local mirror to avoid such issues?
> Error:
> # /usr/bin/dnf --installroot
> /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/
> --releasever 28 --disableplugin=local --setopt=deltarpm=False install
> @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git
> iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw
> make mom openvswitch ovirt-imageio-common policycoreutils-python
> python2-augeas python2-blivet1 python2-coverage python2-dateutil
> python2-dbus python2-decorator python2-devel python2-dmidecode
> python2-enum34 python2-inotify python2-ioprocess python2-ipaddress
> python2-magic python2-mock python2-netaddr python2-pthreading
> python2-pyudev python2-pyyaml python2-requests python2-setuptools
> python2-six python2-subprocess32 python3-PyYAML python3-augeas
> python3-coverage python3-dateutil python3-dbus python3-decorator
> python3-devel python3-magic python3-netaddr python3-nose
> python3-pyudev python3-six python3-yaml rpm-build
> ...
> Transaction Summary
> ================================================================================
> Install 675 Packages
> Total download size: 424 M
> Installed size: 1.5 G
> Downloading Packages:
> ...
> (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00
> (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00
> (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00
> [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
> reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
> reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> Error: Error downloading packages:
> Curl error (28): Timeout was reached for
> http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> Build:
> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[VDSM] Build fail with "Curl error (28): Timeout was reached for..."
by Nir Soffer
Looks like creating mock env fail because of slow network.
Can we have local mirror to avoid such issues?
Error:
# /usr/bin/dnf --installroot
/var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/
--releasever 28 --disableplugin=local --setopt=deltarpm=False install
@buildsys-build autoconf automake createrepo dnf dnf-utils gdb git
iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw
make mom openvswitch ovirt-imageio-common policycoreutils-python
python2-augeas python2-blivet1 python2-coverage python2-dateutil
python2-dbus python2-decorator python2-devel python2-dmidecode
python2-enum34 python2-inotify python2-ioprocess python2-ipaddress
python2-magic python2-mock python2-netaddr python2-pthreading
python2-pyudev python2-pyyaml python2-requests python2-setuptools
python2-six python2-subprocess32 python3-PyYAML python3-augeas
python3-coverage python3-dateutil python3-dbus python3-decorator
python3-devel python3-magic python3-netaddr python3-nose
python3-pyudev python3-six python3-yaml rpm-build
...
Transaction Summary
================================================================================
Install 675 Packages
Total download size: 424 M
Installed size: 1.5 G
Downloading Packages:
...
(530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00
(531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00
(532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00
[MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
seconds]
[FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
seconds]
Error: Error downloading packages:
Curl error (28): Timeout was reached for
http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
seconds]
Build:
https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
5 years, 11 months
Build failed in Jenkins:
system-sync_mirrors-fedora-base-fc29-x86_64 #136
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5211c5a66cf9806dec3d0f314f43ad3524a950c4 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 5211c5a66cf9806dec3d0f314f43ad3524a950c4
Commit message: "jenkins: adding inject mirror for opstool-release"
> git rev-list --no-walk 5211c5a66cf9806dec3d0f314f43ad3524a950c4 # timeout=10
[system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins1193368713129152883.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7'
Build step 'Execute shell' marked build as failure
5 years, 11 months
[JIRA] (OVIRT-2633) Add posibility to remove changes from change
queue before change, which actually fixes the issue
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2633?page=com.atlassian.jir... ]
Barak Korren edited comment on OVIRT-2633 at 12/23/18 9:25 AM:
---------------------------------------------------------------
Not needed - this request seems to be based on an incorrect assumption that the change queue is testing changes one-by-one serially.
In the case mentioned above, if we suppose we already reported P1 as failed, and P10 has been merged, and lets realistically assume it was not merged before the next CQ test started, so what will happen is that P2-P9 will be tested together and fail, and from then it will take Log(2)(8) = 3 cycles to detect and throw away P2. If P10 is merged in the meantime - the next test will test P3-P10 and succeed, passing all changes assuming there are no further regressions.
was (Author: bkorren(a)redhat.com):
Not needed - this request is based on an incorrect assumption that the change queue is testing changes one-by-one serially.
In the case mentioned above, if we suppose we already reported P1 as failed, and P10 has been merged, and lets realistically assume it was not merged before the next CQ test started, so what will happen is that P2-P9 will be tested together and fail, and from then it will take Log(2)(8) = 3 cycles to detect and throw away P2. If P10 is merged in the meantime - the next test will test P3-P10 and succeed, passing all changes assuming there are no further regressions.
> Add posibility to remove changes from change queue before change, which actually fixes the issue
> ------------------------------------------------------------------------------------------------
>
> Key: OVIRT-2633
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2633
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: Change Queue
> Reporter: Martin Perina
> Assignee: infra
>
> Let's have a patch P1, which created an issue and failed verification in change queue. Until a fix is prepared and merged as P10, there were merged other patches P2 - P9. Because P10 is merged as last, it contains all previous changes. And because we already know that P2 - P9 will fail, there is no reason to verify those changes and there should be a way how to remove them from the change queue, so P10 can be verified as soon as possible. Otherwise we are wasting valuable recources.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-2633) Add posibility to remove changes from change
queue before change, which actually fixes the issue
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2633?page=com.atlassian.jir... ]
Barak Korren commented on OVIRT-2633:
-------------------------------------
I've created OVIRT-1852 in the past to try and optimize the CQ algorithm for the case mentioned here.
We could also consider allowing manual removal of changes in case we are certain we don't want to test them - but allowing that may be risky.
> Add posibility to remove changes from change queue before change, which actually fixes the issue
> ------------------------------------------------------------------------------------------------
>
> Key: OVIRT-2633
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2633
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: Change Queue
> Reporter: Martin Perina
> Assignee: infra
>
> Let's have a patch P1, which created an issue and failed verification in change queue. Until a fix is prepared and merged as P10, there were merged other patches P2 - P9. Because P10 is merged as last, it contains all previous changes. And because we already know that P2 - P9 will fail, there is no reason to verify those changes and there should be a way how to remove them from the change queue, so P10 can be verified as soon as possible. Otherwise we are wasting valuable recources.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
[JIRA] (OVIRT-2633) Add posibility to remove changes from change
queue before change, which actually fixes the issue
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2633?page=com.atlassian.jir... ]
Barak Korren updated OVIRT-2633:
--------------------------------
Component/s: Change Queue
> Add posibility to remove changes from change queue before change, which actually fixes the issue
> ------------------------------------------------------------------------------------------------
>
> Key: OVIRT-2633
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2633
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Components: Change Queue
> Reporter: Martin Perina
> Assignee: infra
>
> Let's have a patch P1, which created an issue and failed verification in change queue. Until a fix is prepared and merged as P10, there were merged other patches P2 - P9. Because P10 is merged as last, it contains all previous changes. And because we already know that P2 - P9 will fail, there is no reason to verify those changes and there should be a way how to remove them from the change queue, so P10 can be verified as soon as possible. Otherwise we are wasting valuable recources.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 11 months
