[JIRA] (OVIRT-2631) [VDSM] Build fail with "Curl error (28):
Timeout was reached for..."
by Evgheni Dereveanchin (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2631?page=com.atlassian.jir... ]
Evgheni Dereveanchin commented on OVIRT-2631:
---------------------------------------------
DNF was using the PHX proxy to reach the mirror. In squid logs there's an ABORTED message:
1545307396.649 76049 66.187.230.121 TCP_HIT_ABORTED/200 2650144 GET http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g... - HIER_NONE/- application/x-rpm
This coincides with a time range between 1545307396.649 and 1545307402.447 when there's a lot of different ABORTED messages for both TCP_HIT and TCP_MISS connections (meaning that even stuff that's cached didn't go through)
The second number in the log is strange as it indicates the connection length in seconds and is more than 20 hours while the whole DNF step lasted a mere 10 minutes. Worth updating squid and the OS in general to avoid possible bugs.
I was not able to locate any other cluses on the squid and no traffic spikes on the PHX gateway, however some direct connections to the proxy from bare metals also got cut in this time range. There is no active device between bare metals and the proxy that could cut connections.
Also, this is the only instance of multiple ABORTED messages in logs so may have been caused by some network flop or other conditions. We can monitor this and close if this does not happen again.
> [VDSM] Build fail with "Curl error (28): Timeout was reached for..."
> --------------------------------------------------------------------
>
> Key: OVIRT-2631
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2631
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: Evgheni Dereveanchin
>
> Looks like creating mock env fail because of slow network.
> Can we have local mirror to avoid such issues?
> Error:
> # /usr/bin/dnf --installroot
> /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/
> --releasever 28 --disableplugin=local --setopt=deltarpm=False install
> @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git
> iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw
> make mom openvswitch ovirt-imageio-common policycoreutils-python
> python2-augeas python2-blivet1 python2-coverage python2-dateutil
> python2-dbus python2-decorator python2-devel python2-dmidecode
> python2-enum34 python2-inotify python2-ioprocess python2-ipaddress
> python2-magic python2-mock python2-netaddr python2-pthreading
> python2-pyudev python2-pyyaml python2-requests python2-setuptools
> python2-six python2-subprocess32 python3-PyYAML python3-augeas
> python3-coverage python3-dateutil python3-dbus python3-decorator
> python3-devel python3-magic python3-netaddr python3-nose
> python3-pyudev python3-six python3-yaml rpm-build
> ...
> Transaction Summary
> ================================================================================
> Install 675 Packages
> Total download size: 424 M
> Installed size: 1.5 G
> Downloading Packages:
> ...
> (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00
> (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00
> (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00
> [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
> reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was
> reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> Error: Error downloading packages:
> Curl error (28): Timeout was reached for
> http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/g...
> [Operation too slow. Less than 1000 bytes/sec transferred the last 30
> seconds]
> Build:
> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 10 months
[JIRA] (OVIRT-2636) Tests failed because global_setup.sh failed
by Evgheni Dereveanchin (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2636?page=com.atlassian.jir... ]
Evgheni Dereveanchin commented on OVIRT-2636:
---------------------------------------------
I've offlined the slave to limit further issues yet logging in showed no problems with systemd and the journal was already rotated. As the system had almost a year of uptime I've updated it to the latest 7.5 snapshot we have defined and rebooted it. I also see vm0097 is offline due to journald issues so will repeat the process and schedule replacement of all el7 VMs with fresh 7.6 systems
> Tests failed because global_setup.sh failed
> -------------------------------------------
>
> Key: OVIRT-2636
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2636
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: Evgheni Dereveanchin
>
> Not sure why global setup failed:
> + [[ ! -O /home/jenkins/.ssh ]]
> + [[ ! -G /home/jenkins/.ssh ]]
> + verify_set_permissions 700 /home/jenkins/.ssh
> + local target_permissions=700
> + local path_to_set=/home/jenkins/.ssh
> ++ stat -c %a /home/jenkins/.ssh
> + local access=700
> + [[ 700 != \7\0\0 ]]
> + return 0
> + [[ -f /home/jenkins/.ssh/known_hosts ]]
> + verify_set_ownership /home/jenkins/.ssh/known_hosts
> + local path_to_set=/home/jenkins/.ssh/known_hosts
> ++ id -un
> + local owner=jenkins
> ++ id -gn
> + local group=jenkins
> + [[ ! -O /home/jenkins/.ssh/known_hosts ]]
> + [[ ! -G /home/jenkins/.ssh/known_hosts ]]
> + verify_set_permissions 644 /home/jenkins/.ssh/known_hosts
> + local target_permissions=644
> + local path_to_set=/home/jenkins/.ssh/known_hosts
> ++ stat -c %a /home/jenkins/.ssh/known_hosts
> + local access=644
> + [[ 644 != \6\4\4 ]]
> + return 0
> + return 0
> + true
> + log ERROR Aborting.
> Build:
> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 10 months
[JIRA] (OVIRT-2636) Tests failed because global_setup.sh failed
by Evgheni Dereveanchin (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2636?page=com.atlassian.jir... ]
Evgheni Dereveanchin reassigned OVIRT-2636:
-------------------------------------------
Assignee: Evgheni Dereveanchin (was: infra)
> Tests failed because global_setup.sh failed
> -------------------------------------------
>
> Key: OVIRT-2636
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2636
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: Evgheni Dereveanchin
>
> Not sure why global setup failed:
> + [[ ! -O /home/jenkins/.ssh ]]
> + [[ ! -G /home/jenkins/.ssh ]]
> + verify_set_permissions 700 /home/jenkins/.ssh
> + local target_permissions=700
> + local path_to_set=/home/jenkins/.ssh
> ++ stat -c %a /home/jenkins/.ssh
> + local access=700
> + [[ 700 != \7\0\0 ]]
> + return 0
> + [[ -f /home/jenkins/.ssh/known_hosts ]]
> + verify_set_ownership /home/jenkins/.ssh/known_hosts
> + local path_to_set=/home/jenkins/.ssh/known_hosts
> ++ id -un
> + local owner=jenkins
> ++ id -gn
> + local group=jenkins
> + [[ ! -O /home/jenkins/.ssh/known_hosts ]]
> + [[ ! -G /home/jenkins/.ssh/known_hosts ]]
> + verify_set_permissions 644 /home/jenkins/.ssh/known_hosts
> + local target_permissions=644
> + local path_to_set=/home/jenkins/.ssh/known_hosts
> ++ stat -c %a /home/jenkins/.ssh/known_hosts
> + local access=644
> + [[ 644 != \6\4\4 ]]
> + return 0
> + return 0
> + true
> + log ERROR Aborting.
> Build:
> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 10 months
[JIRA] (OVIRT-2641) mirror_mgr.sh - Build error
by Liora Milbaum (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2641?page=com.atlassian.jir... ]
Liora Milbaum commented on OVIRT-2641:
--------------------------------------
[~bkorren(a)redhat.com] Do we have a DEV environment?
> mirror_mgr.sh - Build error
> ---------------------------
>
> Key: OVIRT-2641
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2641
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: CI Mirrors
> Reporter: Liora Milbaum
> Assignee: infra
>
> From job run:
> [https://jenkins.ovirt.org/blue/organizations/jenkins/system-sync_mirrors-...]
> {code}+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf
> Checking if mirror needs a resync
> Traceback (most recent call last):
> File "/usr/bin/reposync", line 343, in <module>
> main()
> File "/usr/bin/reposync", line 175, in main
> my.doRepoSetup()
> File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
> return self._getRepos(thisrepo, True)
> File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
> self._repos.doSetup(thisrepo)
> File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
> self.retrieveAllMD()
> File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
> dl = repo._async and repo._commonLoadRepoXML(repo)
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
> local = self.cachedir + '/repomd.xml'
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
> cachedir = property(lambda self: self._dirGetAttr('cachedir'))
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
> self.dirSetup()
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
> self._dirSetupMkdir_p(dir)
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
> raise Errors.RepoError, msg
> yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7'
> {code}
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 10 months
Build failed in Jenkins: system-sync_mirrors-epel-el6-x86_64 #2193
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/2193/dis...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-epel-el6-x86_64/ws/>
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 65b7fcabfccd9439d71f60b039775b03d81bf113 (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 65b7fcabfccd9439d71f60b039775b03d81bf113
Commit message: "Add mock configs for Fedora 29"
> git rev-list --no-walk 65b7fcabfccd9439d71f60b039775b03d81bf113 # timeout=10
[system-sync_mirrors-epel-el6-x86_64] $ /bin/bash -xe /tmp/jenkins4569339394683296093.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
local = self.cachedir + '/repomd.xml'
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
cachedir = property(lambda self: self._dirGetAttr('cachedir'))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
self.dirSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
self._dirSetupMkdir_p(dir)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
raise Errors.RepoError, msg
yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7'
Build step 'Execute shell' marked build as failure
5 years, 10 months
[JIRA] (OVIRT-2641) mirror_mgr.sh - Build error
by Liora Milbaum (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2641?page=com.atlassian.jir... ]
Liora Milbaum commented on OVIRT-2641:
--------------------------------------
[~bkorren(a)redhat.com] Correct. Usually it corrects itself in the following run. I will try to look at it.
> mirror_mgr.sh - Build error
> ---------------------------
>
> Key: OVIRT-2641
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2641
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: CI Mirrors
> Reporter: Liora Milbaum
> Assignee: infra
>
> From job run:
> [https://jenkins.ovirt.org/blue/organizations/jenkins/system-sync_mirrors-...]
> {code}+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf
> Checking if mirror needs a resync
> Traceback (most recent call last):
> File "/usr/bin/reposync", line 343, in <module>
> main()
> File "/usr/bin/reposync", line 175, in main
> my.doRepoSetup()
> File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
> return self._getRepos(thisrepo, True)
> File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
> self._repos.doSetup(thisrepo)
> File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
> self.retrieveAllMD()
> File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
> dl = repo._async and repo._commonLoadRepoXML(repo)
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML
> local = self.cachedir + '/repomd.xml'
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda>
> cachedir = property(lambda self: self._dirGetAttr('cachedir'))
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr
> self.dirSetup()
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup
> self._dirSetupMkdir_p(dir)
> File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p
> raise Errors.RepoError, msg
> yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7'
> {code}
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100096)
5 years, 10 months