[OST][hc-basic-suite-master] Fails on HE deployment
by Sahina Bose
xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
772a3dfe-aee8-45d9-9df5-beddcbf92010.ovf\", u'removes': None, u'creates':
None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr':
u\"vdsm-client: Command Image.prepare with args {'storagepoolID':
'e90984b6-af79-11e8-a2dd-00163e24d363', 'storagedomainID':
'b4c84ead-fb7f-4478-acd2-0309d569b9aa', 'volumeID':
'b18f30b1-5625-4581-b815-94291b226c79', 'imageID':
'[u7894c850-665f-47dd-8376-e4da5ae7807f]'} failed:\\n(code=201,
message=Volume does not exist:
(u'b18f30b1-5625-4581-b815-94291b226c79',))\\ntar: This does not look like
a tar archive
Is this a known issue?
thanks!
sahina
6 years, 2 months
PLEASE READ: Stop merging to ovirt-engine master until further notice
by Dafna Ron
Hello to all,
I asked to stop merging last week but it seemed to have been ignored so
writing again.
*please stop merging to ovirt-engine master until further notice.*
the project has been failing for the last two weeks on various different
regressions.
You continue to merge changes which means that you are risking further
regressions and making it hard to stabilize the master branch for
ovirt-engine.
to make it clear:
1. all changes that were merged since sept. 4th to the master branch exist
only in the ovirt-engine master branch (i.e they are not included in your
project package).
2. any code merged to master in the past two weeks was most likely not
tested by CQ since we exit on first test failure (if your code effects
anything after the reported issues ran, they would not be tested).
3. you are putting other projects at risk for regressions since they are
tested with a 2 weeks old ovirt-engine packages.
Thanks you.
Dafna
6 years, 2 months
why we only save lv that seg_start_pe=="0" in _reloadlvs
by pengyixiang
Hello, everyone!
if we reload lvs, why we don't update lvs that seg_start_pe != "0" ? there's in [1] with red mark.
what seg_start_pe means? If this lv is a vm template, may seg_start_pe is not zero, and we
should ignore it?
[1] vdsm/storage/lvm.py
def _reloadlvs(self, vgName, lvNames=None):
lvNames = _normalizeargs(lvNames)
cmd = list(LVS_CMD)
if lvNames:
cmd.extend(["%s/%s" % (vgName, lvName) for lvName in lvNames])
else:
cmd.append(vgName)
rc, out, err = self.cmd(cmd, self._getVGDevs((vgName,)))
with self._lock:
if rc != 0:
log.warning("lvm lvs failed: %s %s %s", str(rc), str(out),
str(err))
lvNames = lvNames if lvNames else self._lvs.keys()
for l in lvNames:
if isinstance(self._lvs.get(l), Stub):
self._lvs[l] = Unreadable(self._lvs[l].name, True)
return dict(self._lvs)
updatedLVs = {}
for line in out:
fields = [field.strip() for field in line.split(SEPARATOR)]
lv = makeLV(*fields)
# For LV we are only interested in its first extent
if lv.seg_start_pe == "0":
self._lvs[(lv.vg_name, lv.name)] = lv
updatedLVs[(lv.vg_name, lv.name)] = lv
# Determine if there are stale LVs
if lvNames:
staleLVs = (lvName for lvName in lvNames
if (vgName, lvName) not in updatedLVs.iterkeys())
else:
# All the LVs in the VG
staleLVs = (lvName for v, lvName in self._lvs.keys()
if (v == vgName) and
((vgName, lvName) not in updatedLVs.iterkeys()))
for lvName in staleLVs:
log.warning("Removing stale lv: %s/%s", vgName, lvName)
self._lvs.pop((vgName, lvName), None)
log.debug("lvs reloaded")
return updatedLVs
--
Cheers，
Pencc
6 years, 2 months
[ OST Failure Report ] [ oVirt Master (vdsm) ] [ 14-08-2018 ] [ 006_migrations.migrate_vm ]
by Dafna Ron
Hi,
We have a failure in vm migration on project vdsm master branch.
I think the issue is related to this change:
https://gerrit.ovirt.org/#/c/94248/ - https://gerrit.ovirt.org/#/c/94248/
Can you please have a look?
full logs:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10309/arti...
error:
2018-09-14 07:15:43,102-0400 ERROR (migsrc/ade7210f) [virt.vm]
(vmId='ade7210f-992e-4256-a33f-1340552892c9') __init__() takes exactly 3
arguments (2 given) (migration:281)
2018-09-14 07:15:43,102-0400 DEBUG (migsrc/ade7210f) [jsonrpc.Notification]
Sending event {"params": {"notify_time": 4296575510,
"ade7210f-992e-4256-a33f-1340552892c9": {"status": "Up"}}, "jsonrpc":
"2.0", "method": "|virt|VM_status|ade72
10f-992e-4256-a33f-1340552892c9"} (__init__:181)
2018-09-14 07:15:43,103-0400 DEBUG (JsonRpc (StompReactor))
[yajsonrpc.protocols.stomp.AsyncClient] Stomp connection established
(stompclient:138)
2018-09-14 07:15:43,104-0400 DEBUG (migsrc/ade7210f) [vds] Sending
notification |virt|VM_status|ade7210f-992e-4256-a33f-1340552892c9 with
params {'notify_time': 4296575510, 'ade7210f-992e-4256-a33f-1340552892c9':
{'status': 'Up'}} (clien
tIF:225)
2018-09-14 07:15:43,104-0400 ERROR (migsrc/ade7210f) [virt.vm]
(vmId='ade7210f-992e-4256-a33f-1340552892c9') Failed to migrate
(migration:449)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 401,
in _regular_run
self._setupVdsConnection()
File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 241,
in _setupVdsConnection
self._destServer = jsonrpcvdscli.connect(requestQueue, client)
File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 259,
in connect
return _Server(client, xml_compat)
File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 135,
in __init__
self._schema = vdsmapi.Schema.vdsm_api(api_strict_mode)
File "/usr/lib/python2.7/site-packages/vdsm/api/vdsmapi.py", line 155, in
vdsm_api
return Schema(schema_types, *args, **kwargs)
TypeError: __init__() takes exactly 3 arguments (2 given)
6 years, 2 months
issue in vdsm monitoring
by Dafna Ron
Hi,
I have been seeing random failures of tests in different projects caused by
vdsm monitoring.
I need someone from vdsm to please help debug this issue.
From what I can see, the test suspend/resume vm failed because we could not
query the status of the vm on the host.
you can see full log from failed tests here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10208/arti...
Here are the errors that I can see in the vdsm which seem to suggest that
there is an issue getting stats on the vm which was suspended and caused
the failure of the test:
2018-09-11 00:40:05,896-0400 INFO (monitor/c1fe6e6)
[storage.StorageDomain] Removing remnants of deleted images [] (fileSD:734)
2018-09-11 00:40:07,957-0400 DEBUG (qgapoller/1) [vds] Not sending QEMU-GA
command 'guest-get-users' to vm_id='8214433a-f233-4aaa-aeda-2ce1d31c78dc',
command is not supported (qemuguestagent:192)
2018-09-11 00:40:08,068-0400 DEBUG (periodic/3)
[virt.sampling.VMBulkstatsMonitor] sampled timestamp 4296118.49 elapsed
0.010 acquired True domains all (sampling:443)
2018-09-11 00:40:08,271-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer]
Calling 'Image.prepare' in bridge with {u'allowIllegal': True,
u'storagepoolID': u'e80a56d9-74da-498a-b010-4a9df287f11d', u'imageID':
u'd4c831e6-02d2-4d89-b516-0ec4597
5e024', u'volumeID': u'15b07af1-625b-42e3-b62a-8e7c7a120a56',
u'storagedomainID': u'f1744940-41b6-4d35-b7bf-870c4e07d995'} (__init__:329)
2018-09-11 00:40:10,846-0400 DEBUG (vmchannels) [virt.vm]
(vmId='8214433a-f233-4aaa-aeda-2ce1d31c78dc') Guest connection timed out
(guestagent:556)
2018-09-11 00:40:11,637-0400 DEBUG (jsonrpc/5) [jsonrpc.JsonRpcServer]
Calling 'Host.getStats' in bridge with {} (__init__:329)
2018-09-11 00:40:11,637-0400 INFO (jsonrpc/5) [api.host] START getStats()
from=::ffff:192.168.201.4,49184 (api:47)
2018-09-11 00:40:11,643-0400 DEBUG (jsonrpc/5) [root] cannot read eth0
speed (nic:42)
2018-09-11 00:40:11,645-0400 DEBUG (jsonrpc/5) [root] cannot read eth1
speed (nic:42)
2018-09-11 00:40:11,647-0400 DEBUG (jsonrpc/5) [root] cannot read eth2
speed (nic:42)
2018-09-11 00:40:11,649-0400 DEBUG (jsonrpc/5) [root] cannot read eth3
speed (nic:42)
2018-09-11 00:40:11,667-0400 INFO (jsonrpc/5) [api.host] FINISH getStats
error=[Errno 22] Invalid argument from=::ffff:192.168.201.4,49184 (api:51)
2018-09-11 00:40:11,667-0400 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer]
Internal server error (__init__:350)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345,
in _handle_request
res = method(**params)
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in
_dynamicMethod
result = fn(*methodArgs)
File "<string>", line 2, in getStats
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 49, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1407, in
getStats
multipath=True)}
File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 49, in
get_stats
decStats = stats.produce(first_sample, last_sample)
File "/usr/lib/python2.7/site-packages/vdsm/host/stats.py", line 71, in
produce
stats.update(get_interfaces_stats())
File "/usr/lib/python2.7/site-packages/vdsm/host/stats.py", line 153, in
get_interfaces_stats
return net_api.network_stats()
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 63, in
network_stats
return netstats.report()
File "/usr/lib/python2.7/site-packages/vdsm/network/netstats.py", line
31, in report
stats = link_stats.report()
File "/usr/lib/python2.7/site-packages/vdsm/network/link/stats.py", line
41, in report
speed = vlan.speed(i.device)
File "/usr/lib/python2.7/site-packages/vdsm/network/link/vlan.py", line
36, in speed
dev_speed = nic.read_speed_using_sysfs(dev_name)
File "/usr/lib/python2.7/site-packages/vdsm/network/link/nic.py", line
48, in read_speed_using_sysfs
s = int(f.read())
IOError: [Errno 22] Invalid argument
2018-09-11 00:40:11,669-0400 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC
call Host.getStats failed (error -32603) in 0.03 seconds (__init__:312)
2018-09-11 00:40:14,656-0400 DEBUG (jsonrpc/2) [jsonrpc.JsonRpcServer]
Calling 'Host.getAllVmStats' in bridge with {} (__init__:329)
2018-09-11 00:40:14,657-0400 INFO (jsonrpc/2) [api.host] START
getAllVmStats() from=::1,46772 (api:47)
2018-09-11 00:40:14,659-0400 INFO (jsonrpc/2) [api.host] FINISH
getAllVmStats return={'status': {'message': 'Done', 'code': 0},
'statsList': (suppressed)} from=::1,46772 (api:53)
2018-09-11 00:40:14,660-0400 DEBUG (jsonrpc/2) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmStats' in bridge with (suppressed) (__init__:356)
2018-09-11 00:40:14,661-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC
call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312)
2018-09-11 00:40:14,674-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer]
Calling 'Host.getAllVmIoTunePolicies' in bridge with {} (__init__:329)
2018-09-11 00:40:14,675-0400 INFO (jsonrpc/1) [api.host] START
getAllVmIoTunePolicies() from=::1,46772 (api:47)
2018-09-11 00:40:14,675-0400 INFO (jsonrpc/1) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'message': 'Done', 'code': 0},
'io_tune_policies_dict': {'8214433a-f233-4aaa-aeda-2ce1d31c78dc':
{'policy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/blockSD/f1744940-41b6-4d35-b7bf-870c4e07d995/images/9a3659dc-339e-4b3f-a1d1-6ee3debb6168/e2570635-a798-4e9e-a0ea-88436d5b296f',
'name': 'vda'}]}}} from=::1,46772 (api:53)
2018-09-11 00:40:14,675-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmIoTunePolicies' in bridge with
{'8214433a-f233-4aaa-aeda-2ce1d31c78dc': {'policy': [], 'current_values':
[{'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec':
0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L},
'path':
'/rhev/data-center/mnt/blockSD/f1744940-41b6-4d35-b7bf-870c4e07d995/images/9a3659dc-339e-4b3f-a1d1-6ee3debb6168/e2570635-a798-4e9e-a0ea-88436d5b296f',
'name': 'vda'}]}} (__init__:356)
2018-09-11 00:40:14,675-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC
call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:312)
2018-09-11 00:40:16,118-0400 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer]
Calling 'Host.getAllVmStats' in bridge with {} (__init__:329)
2018-09-11 00:40:16,119-0400 INFO (jsonrpc/7) [api.host] START
getAllVmStats() from=::ffff:192.168.201.4,49184 (api:47)
2018-09-11 00:40:16,121-0400 INFO (jsonrpc/7) [api.host] FINISH
getAllVmStats return={'status': {'message': 'Done', 'code': 0},
'statsList': (suppressed)} from=::ffff:192.168.201.4,49184 (api:53)
2018-09-11 00:40:16,122-0400 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmStats' in bridge with (suppressed) (__init__:356)
2018-09-11 00:40:16,122-0400 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC
call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312)
2018-09-11 00:40:17,956-0400 DEBUG (qgapoller/2) [vds] Not sending QEMU-GA
command 'guest-get-users' to vm_id='8214433a-f233-4aaa-aeda-2ce1d31c78dc',
command is not supported (qemuguestagent:192)
2018-09-11 00:40:18,025-0400 DEBUG (periodic/3) [virt.periodic] Looking for
stale paused VMs (periodic:388)
I can also see this in Super vdsm which may be related but I an not 100%
sure its related to current issue:
MainProcess|jsonrpc/2::DEBUG::2018-09-11
00:33:20,012::cmdutils::159::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
ifup/on039d2d88d8ee4::DEBUG::2018-09-11
00:33:20,238::cmdutils::159::root::(exec_cmd) FAILED: <err> = 'Running
scope as unit
2f1f417d-85b9-46c0-b7d7-6082836daaae.scope.\n/etc/sysconfig/network-scripts/ifup-eth:
line 321: 20715 Terminated
/sbin/dhclient ${DHCLIENTARGS} ${DEVICE}\nDevice
"on039d2d88d8ee4" does not exist.\nCannot find device
"on039d2d88d8ee4"\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d
8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevic
e "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d
8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevic
e "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d
8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevic
e "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice
"on039d2d88d8ee4" does not
exist.\nDevice "on039d2d88d8ee4" does not exist.\nDevice "on039d2d88d8ee4"
does not exist.\nDevice "on039d2d88d8ee4" does not exist.\n'; <rc> = 1
ifup/on039d2d88d8ee4::ERROR::2018-09-11
00:33:20,238::concurrent::202::root::(run) FINISH thread
<Thread(ifup/on039d2d88d8ee4, stopped daemon 140281550731008)> failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
195, in run
ret = func(*args, **kwargs)
File
"/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py",
line 948, in _exec_ifup
_exec_ifup_by_name(iface.name, cgroup)
File
"/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py",
line 934, in _exec_ifup_by_name
raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
ConfigNetworkError: (29, '\n')
thanks,
Dafna
6 years, 2 months
