[ OST Failure Report ] [ oVirt 4.1 && oVirt master ] [ 20-03-2017 ] [ 004_basic_sanity.snapshots_merge ]
Hi, Test failed: [ 004_basic_sanity.snapshots_merge ] Link to suspected patches: N/A Link to Job: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918 Link to all logs: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/artifact/... 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/artifa... Error snippet from the log: <error> ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at 0x446a758> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within res = func() File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 449, in <lambda> api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok' AttributeError: 'NoneType' object has no attribute 'state' lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 448, in snapshot_live_merge lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within res = func() File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 449, in <lambda> api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok' AttributeError: 'NoneType' object has no attribute 'state' </error> Best Regards, Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) OPEN SOURCE - 1 4 011 && 011 4 1
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi@redhat.com> wrote:
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A Link to Job:
1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918
They look similar, but master has an additional failure (unrelated to the snap merge failure). Allon/Tal - can you have someone from storage look at the failures? Alexander - can you take a look at the below? 2017-03-20 09:30:10,283-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure [GetAllMacPoolRangesByMacPoolId] compiled 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector 2017-03-20 09:30:10,275-04 ERROR [org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.CacheUpdate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not update the Utilization Cache: Error while running SQL query: org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error while running SQL query at org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.runQuery(BaseDao.java:60) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao.getTotalCpuMemCount(HostDwhDao.java:78) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getTotalCpuMemCount(HourlySummaryHelper.java:43) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getCpuMemSummary(HourlySummaryHelper.java:21) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.lookupGlobalUtilization(DashboardDataServlet.java:287) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.getDashboard(DashboardDataServlet.java:261) [frontend.jar:]
1.
Link to all logs:
1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/ test_logs/basic-suite-4.1/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic-suite-4.1/post-004_basic_sanity.py/> 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/>
Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at 0x446a758> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within res = func() File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 449, in <lambda> api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok' AttributeError: 'NoneType' object has no attribute 'state' lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 448, in snapshot_live_merge lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217, in assert_equals_within res = func() File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 449, in <lambda> api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok' AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi@redhat.com>
wrote:
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A
Link to Job: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918
They look similar, but master has an additional failure (unrelated to the snap merge failure). Allon/Tal - can you have someone from storage look at the failures? Alexander - can you take a look at the below?
2017-03-20 09:30:10,283-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJd bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure [GetAllMacPoolRangesByMacPoolId] compiled 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector 2017-03-20 09:30:10,275-04 ERROR [org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.CacheUpd ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not update the Utilization Cache: Error while running SQL query: org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error while running SQL query at org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.runQuery(BaseDao.j ava:60) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao.getTotalCpuMemC ount(HostDwhDao.java:78) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getTotalCp uMemCount(HourlySummaryHelper.java:43) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getCpuMemS ummary(HourlySummaryHelper.java:21) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.lookupGlo balUtilization(DashboardDataServlet.java:287) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.getDashbo ard(DashboardDataServlet.java:261) [frontend.jar:]
So it looks like a communication failure with the DWH. But when I look at the associated logs, the error is not in there, nor do the time stamps match. Is this error from the same run or a previous one? I was trying to get more information than a partial stack trace.
1.
Link to all logs: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/ test_logs/basic-suite-4.1/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/art ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic-suite-4.1/p ost-004_basic_sanity.py/> 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/ artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite -master/post-004_basic_sanity.py/>> Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at 0x446a758>
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state' lago.utils: ERROR: Error while running thread
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
_ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 448, in snapshot_live_merge
lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
It has happened again: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1033/artifact/... Any news concerning this issue? Thanks Gil On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <awels@redhat.com> wrote:
On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi@redhat.com>
wrote:
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A
Link to Job: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918
They look similar, but master has an additional failure (unrelated to the snap merge failure). Allon/Tal - can you have someone from storage look at the failures? Alexander - can you take a look at the below?
2017-03-20 09:30:10,283-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$ PostgresSimpleJd bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure [GetAllMacPoolRangesByMacPoolId] compiled 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll. TagsDirector] (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector 2017-03-20 09:30:10,275-04 ERROR [org.ovirt.engine.ui.frontend.server.dashboard. DashboardDataServlet.CacheUpd ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not update the Utilization Cache: Error while running SQL query: org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error while running SQL query at org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao. runQuery(BaseDao.j ava:60) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.dao. HostDwhDao.getTotalCpuMemC ount(HostDwhDao.java:78) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard. HourlySummaryHelper.getTotalCp uMemCount(HourlySummaryHelper.java:43) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard. HourlySummaryHelper.getCpuMemS ummary(HourlySummaryHelper.java:21) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard. DashboardDataServlet.lookupGlo balUtilization(DashboardDataServlet.java:287) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard. DashboardDataServlet.getDashbo ard(DashboardDataServlet.java:261) [frontend.jar:]
So it looks like a communication failure with the DWH. But when I look at the associated logs, the error is not in there, nor do the time stamps match. Is this error from the same run or a previous one? I was trying to get more information than a partial stack trace.
1.
Link to all logs: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/ test_logs/basic-suite-4.1/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007/art ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/ basic-suite-4.1/p ost-004_basic_sanity.py/> 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918/ artifact/exported-artifacts/basic-suit-master-el7/test_ logs/basic-suite -master/post-004_basic_sanity.py/>> Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at 0x446a758>
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_ basic_sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state' lago.utils: ERROR: Error while running thread
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
_ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_ basic_sanity.py", line 448, in snapshot_live_merge
lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_ basic_sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Looks like the actuall error is in VDSM log: 2017-03-22 11:01:22,410-0400 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='179424c1-b23d-40aa-a091-3c197971f420') Unexpected error (task:871) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3060, in getVolumeInfo volUUID=volUUID).getInfo() File "/usr/share/vdsm/storage/sd.py", line 748, in produceVolume volUUID) File "/usr/share/vdsm/storage/fileVolume.py", line 361, in __init__ manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/share/vdsm/storage/fileVolume.py", line 61, in __init__ volUUID) File "/usr/share/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/share/vdsm/storage/volume.py", line 109, in validate self.validateVolumePath() File "/usr/share/vdsm/storage/fileVolume.py", line 120, in validateVolumePath raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'5133a146-d4bc-49a1-8358-7e14522ffc4b',) 2017-03-22 11:01:22,413-0400 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='179424c1-b23d-40aa-a091-3c197971f420') aborting: Task is aborted: 'Volume does not exist' - code 201 (task:1176) 2017-03-22 11:01:22,414-0400 ERROR (jsonrpc/7) [storage.Dispatcher] {'status': {'message': "Volume does not exist: (u'5133a146-d4bc-49a1-8358-7e14522ffc4b',)", 'code': 201}} (dispatcher:78) On Wed, Mar 22, 2017 at 10:57 AM, Gil Shinar <gshinar@redhat.com> wrote:
It has happened again: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1033/artifact/exported-artifacts/basic-suit-4.1-el7/ test_logs/basic-suite-4.1/post-004_basic_sanity.py/
Any news concerning this issue?
Thanks Gil
On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <awels@redhat.com> wrote:
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi@redhat.com>
wrote:
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A
Link to Job: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4. 1/1007 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918
They look similar, but master has an additional failure (unrelated to
On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote: the
snap merge failure). Allon/Tal - can you have someone from storage look at the failures? Alexander - can you take a look at the below?
2017-03-20 09:30:10,283-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$ PostgresSimpleJd bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure [GetAllMacPoolRangesByMacPoolId] compiled 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.Tag sDirector] (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector 2017-03-20 09:30:10,275-04 ERROR [org.ovirt.engine.ui.frontend.server.dashboard.DashboardData Servlet.CacheUpd ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not update the Utilization Cache: Error while running SQL query: org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error while running SQL query at org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.ru nQuery(BaseDao.j ava:60) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao .getTotalCpuMemC ount(HostDwhDao.java:78) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH elper.getTotalCp uMemCount(HourlySummaryHelper.java:43) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH elper.getCpuMemS ummary(HourlySummaryHelper.java:21) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS ervlet.lookupGlo balUtilization(DashboardDataServlet.java:287) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS ervlet.getDashbo ard(DashboardDataServlet.java:261) [frontend.jar:]
So it looks like a communication failure with the DWH. But when I look at the associated logs, the error is not in there, nor do the time stamps match. Is this error from the same run or a previous one? I was trying to get more information than a partial stack trace.
1.
Link to all logs: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/ test_logs/basic-suite-4.1/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4 .1/1007/art ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic- suite-4.1/p ost-004_basic_sanity.py/> 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_m aster/5918/ artifact/exported-artifacts/basic-suit-master-el7/test_logs/ basic-suite -master/post-004_basic_sanity.py/>> Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at 0x446a758>
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_ sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state' lago.utils: ERROR: Error while running thread
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
_ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_ sanity.py", line 448, in snapshot_live_merge
lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_ sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Hi all, Happened again http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6017/consol... On Wed, Mar 22, 2017 at 4:48 PM, Gil Shinar <gshinar@redhat.com> wrote:
Looks like the actuall error is in VDSM log:
2017-03-22 11:01:22,410-0400 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='179424c1-b23d-40aa-a091-3c197971f420') Unexpected error (task:871) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3060, in getVolumeInfo volUUID=volUUID).getInfo() File "/usr/share/vdsm/storage/sd.py", line 748, in produceVolume volUUID) File "/usr/share/vdsm/storage/fileVolume.py", line 361, in __init__ manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/share/vdsm/storage/fileVolume.py", line 61, in __init__ volUUID) File "/usr/share/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/share/vdsm/storage/volume.py", line 109, in validate self.validateVolumePath() File "/usr/share/vdsm/storage/fileVolume.py", line 120, in validateVolumePath raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'5133a146-d4bc-49a1-8358-7e14522ffc4b',) 2017-03-22 11:01:22,413-0400 INFO (jsonrpc/7) [storage.TaskManager.Task] (Task='179424c1-b23d-40aa-a091-3c197971f420') aborting: Task is aborted: 'Volume does not exist' - code 201 (task:1176) 2017-03-22 11:01:22,414-0400 ERROR (jsonrpc/7) [storage.Dispatcher] {'status': {'message': "Volume does not exist: (u'5133a146-d4bc-49a1-8358-7e14522ffc4b',)", 'code': 201}} (dispatcher:78)
On Wed, Mar 22, 2017 at 10:57 AM, Gil Shinar <gshinar@redhat.com> wrote:
It has happened again: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4. 1/1033/artifact/exported-artifacts/basic-suit-4.1-el7/test_ logs/basic-suite-4.1/post-004_basic_sanity.py/
Any news concerning this issue?
Thanks Gil
On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <awels@redhat.com> wrote:
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi@redhat.com
wrote:
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A
Link to Job: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4. 1/1007 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918
They look similar, but master has an additional failure (unrelated to
On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote: the
snap merge failure). Allon/Tal - can you have someone from storage look at the failures? Alexander - can you take a look at the below?
2017-03-20 09:30:10,283-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$ PostgresSimpleJd bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure [GetAllMacPoolRangesByMacPoolId] compiled 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.Tag sDirector] (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector 2017-03-20 09:30:10,275-04 ERROR [org.ovirt.engine.ui.frontend.server.dashboard.DashboardData Servlet.CacheUpd ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not update the Utilization Cache: Error while running SQL query: org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error while running SQL query at org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.ru nQuery(BaseDao.j ava:60) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao .getTotalCpuMemC ount(HostDwhDao.java:78) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH elper.getTotalCp uMemCount(HourlySummaryHelper.java:43) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH elper.getCpuMemS ummary(HourlySummaryHelper.java:21) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS ervlet.lookupGlo balUtilization(DashboardDataServlet.java:287) [frontend.jar:] at org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS ervlet.getDashbo ard(DashboardDataServlet.java:261) [frontend.jar:]
So it looks like a communication failure with the DWH. But when I look at the associated logs, the error is not in there, nor do the time stamps match. Is this error from the same run or a previous one? I was trying to get more information than a partial stack trace.
1.
Link to all logs: 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/ test_logs/basic-suite-4.1/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4 .1/1007/art ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic- suite-4.1/p ost-004_basic_sanity.py/> 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ master/5918/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-004_basic_sanity.py/ <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_m aster/5918/ artifact/exported-artifacts/basic-suit-master-el7/test_logs/ basic-suite -master/post-004_basic_sanity.py/>> Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at 0x446a758>
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_ sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state' lago.utils: ERROR: Error while running thread
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
_ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_ sanity.py", line 448, in snapshot_live_merge
lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_ sanity.py", line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL RHCSA | RHCVA | RHCE IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Pavel Zhukov Software Engineer RHEV Devops IRC: landgraf
participants (5)
-
Alexander Wels -
Gil Shinar -
Pavel Zhukov -
Shlomo Ben David -
Yaniv Kaul