[ OST Failure Report ] [ oVirt 4.1 && oVirt master ] [ 20-03-2017 ] [ 004_basic_sanity.snapshots_merge ]
9:18 a.m.
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A
Link to Job:
1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007
2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918
Link to all logs:
1.
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/artifa...
2.
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/art...
Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at
0x446a758>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File
"/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
lago.utils: ERROR: Error while running thread
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
_ret_via_queue
queue.put({'return': func()})
File
"/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
line 448, in snapshot_live_merge
lambda:
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File
"/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
RHCSA | RHCVA | RHCE
IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
10:01 a.m.
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi(a)redhat.com>
wrote:
Hi,
Test failed: [ 004_basic_sanity.snapshots_merge ]
Link to suspected patches: N/A
Link to Job:
1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007
2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
master/5918
They look similar, but master has an additional failure (unrelated to the
snap merge failure).
Allon/Tal - can you have someone from storage look at the failures?
Alexander - can you take a look at the below?
2017-03-20 09:30:10,283-04 DEBUG
[org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall]
(ServerService Thread Pool -- 48) [] SqlCall for procedure
[GetAllMacPoolRangesByMacPoolId] compiled
2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.TagsDirector]
(ServerService Thread Pool -- 41) [] Finished initializing TagsDirector
2017-03-20 09:30:10,275-04 ERROR
[org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.CacheUpdate.Utilization]
(EE-ManagedThreadFactory-default-Thread-2) [] Could not update the
Utilization Cache: Error while running SQL query:
org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error
while running SQL query
at
org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.runQuery(BaseDao.java:60)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao.getTotalCpuMemCount(HostDwhDao.java:78)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getTotalCpuMemCount(HourlySummaryHelper.java:43)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getCpuMemSummary(HourlySummaryHelper.java:21)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.lookupGlobalUtilization(DashboardDataServlet.java:287)
[frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.getDashboard(DashboardDataServlet.java:261)
[frontend.jar:]
1.
Link to all logs:
1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/
test_logs/basic-suite-4.1/post-004_basic_sanity.py/
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/artifa...
2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
master/5918/artifact/exported-artifacts/basic-suit-master-
el7/test_logs/basic-suite-master/post-004_basic_sanity.py/
<http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/art...
Error snippet from the log:
<error>
ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at
0x446a758>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
lago.utils: ERROR: Error while running thread
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
_ret_via_queue
queue.put({'return': func()})
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
line 448, in snapshot_live_merge
lambda:
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
in assert_equals_within
res = func()
File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
line 449, in <lambda>
api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
AttributeError: 'NoneType' object has no attribute 'state'
</error>
Best Regards,
Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
RHCSA | RHCVA | RHCE
IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
OPEN SOURCE - 1 4 011 && 011 4 1
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
10:53 a.m.
On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:
On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David
<sbendavi(a)redhat.com>
wrote:
> Hi,
>
>
> Test failed: [ 004_basic_sanity.snapshots_merge ]
>
> Link to suspected patches: N/A
>
> Link to Job:
> 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007
> 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> master/5918
They look similar, but master has an additional failure (unrelated to the
snap merge failure).
Allon/Tal - can you have someone from storage look at the failures?
Alexander - can you take a look at the below?
2017-03-20 09:30:10,283-04 DEBUG
[org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJd
bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure
[GetAllMacPoolRangesByMacPoolId] compiled
2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.TagsDirector]
(ServerService Thread Pool -- 41) [] Finished initializing TagsDirector
2017-03-20 09:30:10,275-04 ERROR
[org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.CacheUpd
ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not
update the Utilization Cache: Error while running SQL query:
org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException: Error
while running SQL query
at
org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.runQuery(BaseDao.j
ava:60) [frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao.getTotalCpuMemC
ount(HostDwhDao.java:78) [frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getTotalCp
uMemCount(HourlySummaryHelper.java:43) [frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryHelper.getCpuMemS
ummary(HourlySummaryHelper.java:21) [frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.lookupGlo
balUtilization(DashboardDataServlet.java:287) [frontend.jar:]
at
org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataServlet.getDashbo
ard(DashboardDataServlet.java:261) [frontend.jar:]
So it looks like a communication failure with the DWH. But when I look at the
associated logs, the error is not in there, nor do the time stamps match. Is
this error from the same run or a previous one? I was trying to get more
information than a partial stack trace.
> 1.
>
> Link to all logs:
> 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/
> test_logs/basic-suite-4.1/post-004_basic_sanity.py/
> <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1007/art
> ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic-suite-4.1/p
> ost-004_basic_sanity.py/> 2.
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> master/5918/artifact/exported-artifacts/basic-suit-master-
> el7/test_logs/basic-suite-master/post-004_basic_sanity.py/
> <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5918/
> artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite
> -master/post-004_basic_sanity.py/>>
> Error snippet from the log:
>
> <error>
>
> ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda> at
> 0x446a758>
>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
>
> in assert_equals_within
>
> res = func()
>
> File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
>
> ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
> line 449, in <lambda>
>
> api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
>
> AttributeError: 'NoneType' object has no attribute 'state'
> lago.utils: ERROR: Error while running thread
>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
>
> _ret_via_queue
>
> queue.put({'return': func()})
>
> File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
>
> ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
> line 448, in snapshot_live_merge
>
> lambda:
> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 264,
>
> in assert_true_within_long
>
> assert_equals_within_long(func, True, allowed_exceptions)
>
> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 251,
>
> in assert_equals_within_long
>
> func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
>
> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 217,
>
> in assert_equals_within
>
> res = func()
>
> File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
>
> ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_sanity.py",
> line 449, in <lambda>
>
> api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
>
> AttributeError: 'NoneType' object has no attribute 'state'
>
> </error>
>
>
> Best Regards,
>
> Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
> RHCSA | RHCVA | RHCE
> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>
> OPEN SOURCE - 1 4 011 && 011 4 1
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
3:57 a.m.
It has happened again:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/1033/artifa...
Any news concerning this issue?
Thanks
Gil
On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <awels(a)redhat.com> wrote:
On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:
> On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi(a)redhat.com>
>
> wrote:
> > Hi,
> >
> >
> > Test failed: [ 004_basic_sanity.snapshots_merge ]
> >
> > Link to suspected patches: N/A
> >
> > Link to Job:
> > 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
4.1/1007
> > 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> > master/5918
>
> They look similar, but master has an additional failure (unrelated to the
> snap merge failure).
> Allon/Tal - can you have someone from storage look at the failures?
> Alexander - can you take a look at the below?
>
>
> 2017-03-20 09:30:10,283-04 DEBUG
> [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$
PostgresSimpleJd
> bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure
> [GetAllMacPoolRangesByMacPoolId] compiled
> 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.
TagsDirector]
> (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector
> 2017-03-20 09:30:10,275-04 ERROR
> [org.ovirt.engine.ui.frontend.server.dashboard.
DashboardDataServlet.CacheUpd
> ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could not
> update the Utilization Cache: Error while running SQL query:
> org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException:
Error
> while running SQL query
> at
> org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.
runQuery(BaseDao.j
> ava:60) [frontend.jar:]
> at
> org.ovirt.engine.ui.frontend.server.dashboard.dao.
HostDwhDao.getTotalCpuMemC
> ount(HostDwhDao.java:78) [frontend.jar:]
> at
> org.ovirt.engine.ui.frontend.server.dashboard.
HourlySummaryHelper.getTotalCp
> uMemCount(HourlySummaryHelper.java:43) [frontend.jar:]
> at
> org.ovirt.engine.ui.frontend.server.dashboard.
HourlySummaryHelper.getCpuMemS
> ummary(HourlySummaryHelper.java:21) [frontend.jar:]
> at
> org.ovirt.engine.ui.frontend.server.dashboard.
DashboardDataServlet.lookupGlo
> balUtilization(DashboardDataServlet.java:287) [frontend.jar:]
> at
> org.ovirt.engine.ui.frontend.server.dashboard.
DashboardDataServlet.getDashbo
> ard(DashboardDataServlet.java:261) [frontend.jar:]
>
So it looks like a communication failure with the DWH. But when I look at
the
associated logs, the error is not in there, nor do the time stamps match.
Is
this error from the same run or a previous one? I was trying to get more
information than a partial stack trace.
> > 1.
> >
> > Link to all logs:
> > 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> > 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/
> > test_logs/basic-suite-4.1/post-004_basic_sanity.py/
> > <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
4.1/1007/art
> > ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/
basic-suite-4.1/p
> > ost-004_basic_sanity.py/> 2.
> > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> > master/5918/artifact/exported-artifacts/basic-suit-master-
> > el7/test_logs/basic-suite-master/post-004_basic_sanity.py/
> > <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
master/5918/
> > artifact/exported-artifacts/basic-suit-master-el7/test_
logs/basic-suite
> > -master/post-004_basic_sanity.py/>>
> > Error snippet from the log:
> >
> > <error>
> >
> > ovirtlago.testlib: ERROR: Unhandled exception in <function <lambda>
at
> > 0x446a758>
> >
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
217,
> >
> > in assert_equals_within
> >
> > res = func()
> >
> > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
> >
> > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_
basic_sanity.py",
> > line 449, in <lambda>
> >
> > api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
> >
> > AttributeError: 'NoneType' object has no attribute 'state'
> > lago.utils: ERROR: Error while running thread
> >
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in
> >
> > _ret_via_queue
> >
> > queue.put({'return': func()})
> >
> > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
> >
> > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_
basic_sanity.py",
> > line 448, in snapshot_live_merge
> >
> > lambda:
> > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
264,
> >
> > in assert_true_within_long
> >
> > assert_equals_within_long(func, True, allowed_exceptions)
> >
> > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
251,
> >
> > in assert_equals_within_long
> >
> > func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
> >
> > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
217,
> >
> > in assert_equals_within
> >
> > res = func()
> >
> > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
> >
> > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_
basic_sanity.py",
> > line 449, in <lambda>
> >
> > api.vms.get(VM0_NAME).disks.get(disk_name).status.state == 'ok'
> >
> > AttributeError: 'NoneType' object has no attribute 'state'
> >
> > </error>
> >
> >
> > Best Regards,
> >
> > Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
> > RHCSA | RHCVA | RHCE
> > IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
> >
> > OPEN SOURCE - 1 4 011 && 011 4 1
> >
> >
> > _______________________________________________
> > Devel mailing list
> > Devel(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
10:48 a.m.
Looks like the actuall error is in VDSM log:
2017-03-22 11:01:22,410-0400 ERROR (jsonrpc/7)
[storage.TaskManager.Task]
(Task='179424c1-b23d-40aa-a091-3c197971f420') Unexpected error
(task:871)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
878, in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3060, in getVolumeInfo
volUUID=volUUID).getInfo()
File "/usr/share/vdsm/storage/sd.py", line 748, in produceVolume
volUUID)
File "/usr/share/vdsm/storage/fileVolume.py", line 361, in __init__
manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/fileVolume.py", line 61, in __init__
volUUID)
File "/usr/share/vdsm/storage/volume.py", line 86, in __init__
self.validate()
File "/usr/share/vdsm/storage/volume.py", line 109, in validate
self.validateVolumePath()
File "/usr/share/vdsm/storage/fileVolume.py", line 120, in validateVolumePath
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'5133a146-d4bc-49a1-8358-7e14522ffc4b',)
2017-03-22 11:01:22,413-0400 INFO (jsonrpc/7)
[storage.TaskManager.Task]
(Task='179424c1-b23d-40aa-a091-3c197971f420') aborting: Task is
aborted: 'Volume does not exist' - code 201 (task:1176)
2017-03-22 11:01:22,414-0400 ERROR (jsonrpc/7) [storage.Dispatcher]
{'status': {'message': "Volume does not exist:
(u'5133a146-d4bc-49a1-8358-7e14522ffc4b',)", 'code': 201}}
(dispatcher:78)
On Wed, Mar 22, 2017 at 10:57 AM, Gil Shinar <gshinar(a)redhat.com> wrote:
It has happened again:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
4.1/1033/artifact/exported-artifacts/basic-suit-4.1-el7/
test_logs/basic-suite-4.1/post-004_basic_sanity.py/
Any news concerning this issue?
Thanks
Gil
On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <awels(a)redhat.com> wrote:
> On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:
> > On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi(a)redhat.com>
> >
> > wrote:
> > > Hi,
> > >
> > >
> > > Test failed: [ 004_basic_sanity.snapshots_merge ]
> > >
> > > Link to suspected patches: N/A
> > >
> > > Link to Job:
> > > 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.
> 1/1007
> > > 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> > > master/5918
> >
> > They look similar, but master has an additional failure (unrelated to
> the
> > snap merge failure).
> > Allon/Tal - can you have someone from storage look at the failures?
> > Alexander - can you take a look at the below?
> >
> >
> > 2017-03-20 09:30:10,283-04 DEBUG
> > [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$
> PostgresSimpleJd
> > bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure
> > [GetAllMacPoolRangesByMacPoolId] compiled
> > 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.Tag
> sDirector]
> > (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector
> > 2017-03-20 09:30:10,275-04 ERROR
> > [org.ovirt.engine.ui.frontend.server.dashboard.DashboardData
> Servlet.CacheUpd
> > ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could
> not
> > update the Utilization Cache: Error while running SQL query:
> > org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException:
> Error
> > while running SQL query
> > at
> > org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.ru
> nQuery(BaseDao.j
> > ava:60) [frontend.jar:]
> > at
> > org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao
> .getTotalCpuMemC
> > ount(HostDwhDao.java:78) [frontend.jar:]
> > at
> > org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH
> elper.getTotalCp
> > uMemCount(HourlySummaryHelper.java:43) [frontend.jar:]
> > at
> > org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH
> elper.getCpuMemS
> > ummary(HourlySummaryHelper.java:21) [frontend.jar:]
> > at
> > org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS
> ervlet.lookupGlo
> > balUtilization(DashboardDataServlet.java:287) [frontend.jar:]
> > at
> > org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS
> ervlet.getDashbo
> > ard(DashboardDataServlet.java:261) [frontend.jar:]
> >
>
> So it looks like a communication failure with the DWH. But when I look at
> the
> associated logs, the error is not in there, nor do the time stamps match.
> Is
> this error from the same run or a previous one? I was trying to get more
> information than a partial stack trace.
>
> > > 1.
> > >
> > > Link to all logs:
> > > 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> > > 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/
> > > test_logs/basic-suite-4.1/post-004_basic_sanity.py/
> > > <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4
> .1/1007/art
> > > ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic-
> suite-4.1/p
> > > ost-004_basic_sanity.py/> 2.
> > > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
> > > master/5918/artifact/exported-artifacts/basic-suit-master-
> > > el7/test_logs/basic-suite-master/post-004_basic_sanity.py/
> > > <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_m
> aster/5918/
> > > artifact/exported-artifacts/basic-suit-master-el7/test_logs/
> basic-suite
> > > -master/post-004_basic_sanity.py/>>
> > > Error snippet from the log:
> > >
> > > <error>
> > >
> > > ovirtlago.testlib: ERROR: Unhandled exception in <function
<lambda> at
> > > 0x446a758>
> > >
> > > Traceback (most recent call last):
> > > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
line
> 217,
> > >
> > > in assert_equals_within
> > >
> > > res = func()
> > >
> > > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
> > >
> > > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_
> sanity.py",
> > > line 449, in <lambda>
> > >
> > > api.vms.get(VM0_NAME).disks.get(disk_name).status.state ==
'ok'
> > >
> > > AttributeError: 'NoneType' object has no attribute 'state'
> > > lago.utils: ERROR: Error while running thread
> > >
> > > Traceback (most recent call last):
> > > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57,
in
> > >
> > > _ret_via_queue
> > >
> > > queue.put({'return': func()})
> > >
> > > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
> > >
> > > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_
> sanity.py",
> > > line 448, in snapshot_live_merge
> > >
> > > lambda:
> > > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
line
> 264,
> > >
> > > in assert_true_within_long
> > >
> > > assert_equals_within_long(func, True, allowed_exceptions)
> > >
> > > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
line
> 251,
> > >
> > > in assert_equals_within_long
> > >
> > > func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
> > >
> > > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
line
> 217,
> > >
> > > in assert_equals_within
> > >
> > > res = func()
> > >
> > > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
> > >
> > > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_
> sanity.py",
> > > line 449, in <lambda>
> > >
> > > api.vms.get(VM0_NAME).disks.get(disk_name).status.state ==
'ok'
> > >
> > > AttributeError: 'NoneType' object has no attribute 'state'
> > >
> > > </error>
> > >
> > >
> > > Best Regards,
> > >
> > > Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
> > > RHCSA | RHCVA | RHCE
> > > IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
> > >
> > > OPEN SOURCE - 1 4 011 && 011 4 1
> > >
> > >
> > > _______________________________________________
> > > Devel mailing list
> > > Devel(a)ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
2:32 a.m.
Hi all,
Happened again
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6017/con...
On Wed, Mar 22, 2017 at 4:48 PM, Gil Shinar <gshinar(a)redhat.com> wrote:
Looks like the actuall error is in VDSM log:
2017-03-22 11:01:22,410-0400 ERROR (jsonrpc/7) [storage.TaskManager.Task]
(Task='179424c1-b23d-40aa-a091-3c197971f420') Unexpected error (task:871)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in
wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3060, in getVolumeInfo
volUUID=volUUID).getInfo()
File "/usr/share/vdsm/storage/sd.py", line 748, in produceVolume
volUUID)
File "/usr/share/vdsm/storage/fileVolume.py", line 361, in __init__
manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/fileVolume.py", line 61, in __init__
volUUID)
File "/usr/share/vdsm/storage/volume.py", line 86, in __init__
self.validate()
File "/usr/share/vdsm/storage/volume.py", line 109, in validate
self.validateVolumePath()
File "/usr/share/vdsm/storage/fileVolume.py", line 120, in
validateVolumePath
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'5133a146-d4bc-49a1-8358-7e14522ffc4b',)
2017-03-22 11:01:22,413-0400 INFO (jsonrpc/7) [storage.TaskManager.Task]
(Task='179424c1-b23d-40aa-a091-3c197971f420') aborting: Task is aborted:
'Volume does not exist' - code 201 (task:1176)
2017-03-22 11:01:22,414-0400 ERROR (jsonrpc/7) [storage.Dispatcher] {'status':
{'message': "Volume does not exist:
(u'5133a146-d4bc-49a1-8358-7e14522ffc4b',)", 'code': 201}}
(dispatcher:78)
On Wed, Mar 22, 2017 at 10:57 AM, Gil Shinar <gshinar(a)redhat.com> wrote:
> It has happened again:
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.
> 1/1033/artifact/exported-artifacts/basic-suit-4.1-el7/test_
> logs/basic-suite-4.1/post-004_basic_sanity.py/
>
> Any news concerning this issue?
>
> Thanks
> Gil
>
> On Mon, Mar 20, 2017 at 5:53 PM, Alexander Wels <awels(a)redhat.com> wrote:
>
>> On Monday, March 20, 2017 11:01:21 AM EDT Yaniv Kaul wrote:
>> > On Mon, Mar 20, 2017 at 4:18 PM, Shlomo Ben David <sbendavi(a)redhat.com
>> >
>> >
>> > wrote:
>> > > Hi,
>> > >
>> > >
>> > > Test failed: [ 004_basic_sanity.snapshots_merge ]
>> > >
>> > > Link to suspected patches: N/A
>> > >
>> > > Link to Job:
>> > > 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.
>> 1/1007
>> > > 2. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>> > > master/5918
>> >
>> > They look similar, but master has an additional failure (unrelated to
>> the
>> > snap merge failure).
>> > Allon/Tal - can you have someone from storage look at the failures?
>> > Alexander - can you take a look at the below?
>> >
>> >
>> > 2017-03-20 09:30:10,283-04 DEBUG
>> > [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$
>> PostgresSimpleJd
>> > bcCall] (ServerService Thread Pool -- 48) [] SqlCall for procedure
>> > [GetAllMacPoolRangesByMacPoolId] compiled
>> > 2017-03-20 09:30:10,283-04 INFO [org.ovirt.engine.core.bll.Tag
>> sDirector]
>> > (ServerService Thread Pool -- 41) [] Finished initializing TagsDirector
>> > 2017-03-20 09:30:10,275-04 ERROR
>> > [org.ovirt.engine.ui.frontend.server.dashboard.DashboardData
>> Servlet.CacheUpd
>> > ate.Utilization] (EE-ManagedThreadFactory-default-Thread-2) [] Could
>> not
>> > update the Utilization Cache: Error while running SQL query:
>> > org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataException:
>> Error
>> > while running SQL query
>> > at
>> > org.ovirt.engine.ui.frontend.server.dashboard.dao.BaseDao.ru
>> nQuery(BaseDao.j
>> > ava:60) [frontend.jar:]
>> > at
>> > org.ovirt.engine.ui.frontend.server.dashboard.dao.HostDwhDao
>> .getTotalCpuMemC
>> > ount(HostDwhDao.java:78) [frontend.jar:]
>> > at
>> > org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH
>> elper.getTotalCp
>> > uMemCount(HourlySummaryHelper.java:43) [frontend.jar:]
>> > at
>> > org.ovirt.engine.ui.frontend.server.dashboard.HourlySummaryH
>> elper.getCpuMemS
>> > ummary(HourlySummaryHelper.java:21) [frontend.jar:]
>> > at
>> > org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS
>> ervlet.lookupGlo
>> > balUtilization(DashboardDataServlet.java:287) [frontend.jar:]
>> > at
>> > org.ovirt.engine.ui.frontend.server.dashboard.DashboardDataS
>> ervlet.getDashbo
>> > ard(DashboardDataServlet.java:261) [frontend.jar:]
>> >
>>
>> So it looks like a communication failure with the DWH. But when I look
>> at the
>> associated logs, the error is not in there, nor do the time stamps
>> match. Is
>> this error from the same run or a previous one? I was trying to get more
>> information than a partial stack trace.
>>
>> > > 1.
>> > >
>> > > Link to all logs:
>> > > 1. http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>> > > 4.1/1007/artifact/exported-artifacts/basic-suit-4.1-el7/
>> > > test_logs/basic-suite-4.1/post-004_basic_sanity.py/
>> > > <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4
>> .1/1007/art
>> > > ifact/exported-artifacts/basic-suit-4.1-el7/test_logs/basic-
>> suite-4.1/p
>> > > ost-004_basic_sanity.py/> 2.
>> > > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>> > > master/5918/artifact/exported-artifacts/basic-suit-master-
>> > > el7/test_logs/basic-suite-master/post-004_basic_sanity.py/
>> > > <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_m
>> aster/5918/
>> > > artifact/exported-artifacts/basic-suit-master-el7/test_logs/
>> basic-suite
>> > > -master/post-004_basic_sanity.py/>>
>> > > Error snippet from the log:
>> > >
>> > > <error>
>> > >
>> > > ovirtlago.testlib: ERROR: Unhandled exception in <function
<lambda>
>> at
>> > > 0x446a758>
>> > >
>> > > Traceback (most recent call last):
>> > > File
"/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
>> line 217,
>> > >
>> > > in assert_equals_within
>> > >
>> > > res = func()
>> > >
>> > > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
>> > >
>> > > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_
>> sanity.py",
>> > > line 449, in <lambda>
>> > >
>> > > api.vms.get(VM0_NAME).disks.get(disk_name).status.state ==
'ok'
>> > >
>> > > AttributeError: 'NoneType' object has no attribute
'state'
>> > > lago.utils: ERROR: Error while running thread
>> > >
>> > > Traceback (most recent call last):
>> > > File "/usr/lib/python2.7/site-packages/lago/utils.py", line
57, in
>> > >
>> > > _ret_via_queue
>> > >
>> > > queue.put({'return': func()})
>> > >
>> > > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
>> > >
>> > > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_
>> sanity.py",
>> > > line 448, in snapshot_live_merge
>> > >
>> > > lambda:
>> > > File
"/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
>> line 264,
>> > >
>> > > in assert_true_within_long
>> > >
>> > > assert_equals_within_long(func, True, allowed_exceptions)
>> > >
>> > > File
"/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
>> line 251,
>> > >
>> > > in assert_equals_within_long
>> > >
>> > > func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
>> > >
>> > > File
"/usr/lib/python2.7/site-packages/ovirtlago/testlib.py",
>> line 217,
>> > >
>> > > in assert_equals_within
>> > >
>> > > res = func()
>> > >
>> > > File "/home/jenkins/workspace/test-repo_ovirt_experimental_4.1/
>> > >
>> > > ovirt-system-tests/basic-suite-4.1/test-scenarios/004_basic_
>> sanity.py",
>> > > line 449, in <lambda>
>> > >
>> > > api.vms.get(VM0_NAME).disks.get(disk_name).status.state ==
'ok'
>> > >
>> > > AttributeError: 'NoneType' object has no attribute
'state'
>> > >
>> > > </error>
>> > >
>> > >
>> > > Best Regards,
>> > >
>> > > Shlomi Ben-David | Software Engineer | Red Hat ISRAEL
>> > > RHCSA | RHCVA | RHCE
>> > > IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>> > >
>> > > OPEN SOURCE - 1 4 011 && 011 4 1
>> > >
>> > >
>> > > _______________________________________________
>> > > Devel mailing list
>> > > Devel(a)ovirt.org
>> > > http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--
Pavel Zhukov
Software Engineer
RHEV Devops
IRC: landgraf
2580
days inactive
2587
days old
5 comments
5 participants
participants (5)
-
Alexander Wels -
Gil Shinar -
Pavel Zhukov -
Shlomo Ben David -
Yaniv Kaul