January 2020 - Infra - oVirt List Archives

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #3178
by jenkins＠jenkins.phx.ovirt.org 06 Jan '20

06 Jan '20

See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 0a841e428861163f8ceafc91394e555d6ffd7352 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 0a841e428861163f8ceafc91394e555d6ffd7352 Commit message: "decorate.py: Added secret decryption" > git rev-list --no-walk 0a841e428861163f8ceafc91394e555d6ffd7352 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins8931590272765583077.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=centos-kvm-common-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs centos-kvm-common-el7 yum + local repo_name=centos-kvm-common-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/centos-kvm-common-el7 /var/www/html/repos/yum/centos-kvm-common-el7/base + check_yum_sync_needed centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=centos-kvm-common-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/centos-kvm-common-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=centos-kvm-common-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=centos-kvm-common-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/centos-kvm-common-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml does not exist + reposync_out='Could not parse metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64 error was File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml.tmp is not XML' Build step 'Execute shell' marked build as failure

1 1

[CQ]: 105533, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 06 Jan '20

06 Jan '20

A system test invoked by the "ovirt-master" change queue including change 105533,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 105994,5 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 105994,5 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/105533/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/105994/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17930/

1 0

[CQ]: 105994,5 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 06 Jan '20

06 Jan '20

Change 105994,5 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/105994/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17928/

1 0

[CQ]: 106029, 5 (ovirt-hosted-engine-setup) failed "ovirt-master" system tests
by oVirt Jenkins 06 Jan '20

06 Jan '20

Change 106029,5 (ovirt-hosted-engine-setup) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/106029/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17926/

1 0

[JIRA] (OVIRT-2852) OST is failing - Last successful run was Dec-13-2019
by Amit Bawer (oVirt JIRA) 06 Jan '20

06 Jan '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2852?page=com.atlassian.jira.… ] Amit Bawer commented on OVIRT-2852: ----------------------------------- AF On Mon, Jan 6, 2020 at 11:48 AM Martin Perina <mperina(a)redhat.com> wrote: > > > On Sun, Jan 5, 2020 at 10:08 AM Amit Bawer <abawer(a)redhat.com> wrote: > >> Seems we have NFS permissions issue for el8 vdsm in some of the runs. >> >> Example from >> https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/arti… >> : >> >> >> 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am >> the actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1 >> (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)... >> 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error >> checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata >> (monitor:501) >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line >> 499, in _pathChecked >> delay = result.delay() >> File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line >> 391, in delay >> raise exception.MiscFileReadException(self.path, self.rc, self.err) >> vdsm.storage.exception.MiscFileReadException: Internal file read failure: >> ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata', >> 1, bytearray(b"/usr/bin/dd: failed to open >> \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\': >> Operation not permitted\n")) >> 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] >> Calling 'StoragePool.disconnect' in bridge with {'storagepoolID': >> 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey': >> 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (__init__:329) >> 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START >> disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439', >> hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786, >> flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48) >> >> >> Probably need to set NFS server export options as in >> https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7 >> > > Here is fix for NFS server on EL8: https://gerrit.ovirt.org/106120 > > Should this be changes also for NFS server on EL7? > AFAICT the need to change the NFS options mostly arises from changes in vdsm dependencies for el8, such as libvirt, requiring a different access for NFS shares than before. So if we are testing el8 hosts with el7 NFS server that might be relvent for el7 NFS server as well. > _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WLLVWPZU6HFDW… >> > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. > > OST is failing - Last successful run was Dec-13-2019 > ---------------------------------------------------- > > Key: OVIRT-2852 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2852 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Amit Bawer > Assignee: infra > > Seems we have NFS permissions issue for el8 vdsm in some of the runs. > Example from > https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/arti… > : > 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the > actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1 > (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)... > 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error > checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata > (monitor:501) > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line > 499, in _pathChecked > delay = result.delay() > File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, > in delay > raise exception.MiscFileReadException(self.path, self.rc, self.err) > vdsm.storage.exception.MiscFileReadException: Internal file read failure: > ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata', > 1, bytearray(b"/usr/bin/dd: failed to open > \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\': > Operation not permitted\n")) > 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] > Calling 'StoragePool.disconnect' in bridge with {'storagepoolID': > 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey': > 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (__init__:329) > 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START > disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439', > hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786, > flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48) > Probably need to set NFS server export options as in > https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100118)

1 0

[JIRA] (OVIRT-2852) OST is failing - Last successful run was Dec-13-2019
by Martin Perina (oVirt JIRA) 06 Jan '20

06 Jan '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2852?page=com.atlassian.jira.… ] Martin Perina commented on OVIRT-2852: -------------------------------------- On Sun, Jan 5, 2020 at 10:08 AM Amit Bawer <abawer(a)redhat.com> wrote: > Seems we have NFS permissions issue for el8 vdsm in some of the runs. > > Example from > https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/arti… > : > > > 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the > actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1 > (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)... > 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error > checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata > (monitor:501) > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line > 499, in _pathChecked > delay = result.delay() > File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, > in delay > raise exception.MiscFileReadException(self.path, self.rc, self.err) > vdsm.storage.exception.MiscFileReadException: Internal file read failure: > ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata', > 1, bytearray(b"/usr/bin/dd: failed to open > \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\': > Operation not permitted\n")) > 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] > Calling 'StoragePool.disconnect' in bridge with {'storagepoolID': > 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey': > 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (__init__:329) > 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START > disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439', > hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786, > flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48) > > > Probably need to set NFS server export options as in > https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7 > Here is fix for NFS server on EL8: https://gerrit.ovirt.org/106120 Should this be changes also for NFS server on EL7? _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WLLVWPZU6HFDW… > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o. > OST is failing - Last successful run was Dec-13-2019 > ---------------------------------------------------- > > Key: OVIRT-2852 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2852 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Amit Bawer > Assignee: infra > > Seems we have NFS permissions issue for el8 vdsm in some of the runs. > Example from > https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/arti… > : > 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the > actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1 > (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)... > 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error > checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata > (monitor:501) > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line > 499, in _pathChecked > delay = result.delay() > File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, > in delay > raise exception.MiscFileReadException(self.path, self.rc, self.err) > vdsm.storage.exception.MiscFileReadException: Internal file read failure: > ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata', > 1, bytearray(b"/usr/bin/dd: failed to open > \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\': > Operation not permitted\n")) > 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] > Calling 'StoragePool.disconnect' in bridge with {'storagepoolID': > 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey': > 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (__init__:329) > 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START > disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439', > hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786, > flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48) > Probably need to set NFS server export options as in > https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7 -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100118)

1 0

[CQ]: 106041, 3 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 06 Jan '20

06 Jan '20

A system test invoked by the "ovirt-master" change queue including change 106041,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 106038,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 106038,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/106041/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/106038/2 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17917/

1 0

oVirt infra daily report - unstable production jobs - 1061
by jenkins＠jenkins.phx.ovirt.org 06 Jan '20

06 Jan '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1061//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 106040, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 05 Jan '20

05 Jan '20

A system test invoked by the "ovirt-master" change queue including change 106040,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 106038,2 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 106038,2 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/106040/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/106038/2 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17915/

1 0

[CQ]: 106038,2 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 05 Jan '20

05 Jan '20

Change 106038,2 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/106038/2 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17913/

1 0