1 p.m.
Hi team,
We have installed ovirt 4.4.
We have a self hosted Engine setup in the environment which has 1 hosted engine on top of
1 deployment-host.
Goal:
We want to create a storage domain of POSIX compliant type for mounting a ceph based
infrastructure.
We have done SRV based resolution in our DNS server but we are unable to create a storage
domain.
Issue:
We are passing the following information:
path: :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
vfs-type: ceph
mounting option:rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA=
We get the following errors:
====================vdsm.log==================================
2023-04-20 11:26:30,318+0530 INFO (jsonrpc/7) [storage.Mount] mounting
:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 at /rhev/data-center/mnt/
:_volumes_xyz_conf_2ee9c2d0-873b-4d04-8c46-4c0da02787b8 (mount:207)
2023-04-20 11:26:30,384+0530 ERROR (jsonrpc/7) [storage.HSM] Could not connect to
storageServer (hsm:2374)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2371, in
connectStorageServer
conObj.connect()
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
180, in connect
six.reraise(t, v, tb)
File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
171, in connect
self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 210, in
mount
cgroup=cgroup)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>
**kwargs)
File "<string>", line 2, in mount
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in
_callmethod
raise convert_to_error(kind, result)
vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t',
'ceph', '-o',
'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', '
:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', '/rhev/data-center/mnt/
:_volumes_xyz_conf_2ee9c2d0-873b-4d04-8c46-4c0da02787b8'] failed with rc=32
out=b'' err=b'mount error 3 = No such process\n'
2023-04-20 11:31:05,715+0530 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH
connectStorageServer error=:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 is not
a valid hosttail address: (dispatcher:87)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/common/network/address.py", line
42, in hosttail_split
raise ValueError
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 74,
in wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 110, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1190, in
prepare
raise self.error
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in
_run
return fn(*args, **kargs)
File "<decorator-gen-117>", line 2, in connectStorageServer
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2368, in
connectStorageServer
conObj = storageServer.ConnectionFactory.createConnection(conInfo)
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
741, in createConnection
return ctor(**params)
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
154, in __init__
self._remotePath = fileUtils.normalize_path(spec)
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileUtils.py", line 98, in
normalize_path
host, tail = address.hosttail_split(path)
File "/usr/lib/python3.6/site-packages/vdsm/common/network/address.py", line
45, in hosttail_split
raise HosttailError('%s is not a valid hosttail address:' % hosttail)
vdsm.common.network.address.HosttailError:
:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 is not a valid hosttail address:
2023-04-20 11:31:05,715+0530 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
StoragePool.connectStorageServer failed (error 451) in 0.00 seconds (__init__:312)
====================engine.log================================
2023-04-20 11:31:03,818+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] HostName = deployment-host
2023-04-20 11:31:03,818+05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Command
'ConnectStorageServerVDSCommand(HostName = deployment-host,
StorageServerConnectionManagementVDSParameters:{hostId='745b7584-0a43-47d9-985f-af0a0155e787',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='POSIXFS',
connectionList='[StorageServerConnections:{id='null',
connection=':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8',
iqn='null', vfsType='ceph',
mountOptions='rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'})' execution failed: VDSGenericException:
VDSErrorException: Failed to ConnectStorageServerVDS, error = Error storage server
connection: ("domType=6, spUUID=00000000-0000-0000-0000-000000000000,
conList=[{'password': '********', 'vfs_type'
: 'ceph', 'port': '', 'mnt_options':
'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', 'iqn':
'', 'connection':
':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', 'ipv6_enabled':
'false', 'id': '00000000-0000-0000-0000-000000000000',
'user': '', 'tpgt': '1'}]",), code = 451
2023-04-20 11:31:03,818+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] FINISH, ConnectStorageServerVDSCommand,
return: , log id: 19c06f3f
2023-04-20 11:31:03,819+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'
failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to ConnectStorageServerVDS, error = Error
storage server connection: ("domType=6, spUUID=00000000-0000-0000-0000-000000000000,
conList=[{'password': '********', 'vfs_type': 'ceph',
'port': '', 'mnt_options':
'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', 'iqn':
'', 'connection':
':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', 'ipv6_enabled':
'false', 'id': '00000000-0000-0000-0000-000000000000',
'user': '', 'tpgt': '1'}]",), code = 451 (Failed with
error StorageServerConnectionError and code 451)
2023-04-20 11:31:03,824+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Transaction rolled-back for command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'.
2023-04-20 11:31:03,847+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Lock freed to object
'EngineLock:{exclusiveLocks='[:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8=STORAGE_CONNECTION]',
sharedLocks=''}'
2023-04-20 11:34:27,788+05 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [63fbb9de] Lock
Acquired to object
'EngineLock:{exclusiveLocks='[c6e0aa38-20ff-4be3-9b34-81a7b0fabb6a=PROVIDER]',
sharedLocks=''}'
2023-04-20 11:34:27,809+05 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [63fbb9de]
Running command: SyncNetworkProviderCommand internal: true.
2023-04-20 11:34:28,017+05 INFO [org.ovirt.engine.core.sso.service.AuthenticationService]
(default task-29) [] User admin@internal-authz with profile [internal] successfully logged
in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search
ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate
ovirt-ext=token:password-access
===============================================================================
We tried creating a NFS type storage domain using the hostname of the nfs server. This
hostname is resolved using the dns server. We are able to create this storage domain.
We also tried creating the POSIX compliant domain using the hostname of the ceph-mon node
but it is failing.
We are passing the following information:
path: ceph-node2:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
vfs-type: ceph
mounting option:rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA=
In this scenario though the mounting is happening but the storage domain is not created
correctly.
we are getting the following errors:
====================vdsm.log==================================
2023-04-20 12:23:54,759+0530 INFO (jsonrpc/0) [api.virt] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'},
'statsList': [{'statusTime': '3009297867', 'status':
'Up', 'vmId': '9ea50595-33cc-4fde-9cd2-ac35c4e97a2c',
'vmName': 'HostedEngine', 'vmType': 'kvm',
'kvmEnable': 'true', 'acpiEnable': 'true',
'elapsedTime': '71325', 'monitorResponse': '0',
'clientIp': '', 'timeOffset': '0', 'pauseCode':
'NOERR', 'cpuUser': '7.12', 'cpuSys': '1.80',
'cpuUsage': '1264730000000', 'network': {'vnet1':
{'macAddr': '00:16:3e:54:7f:23', 'name': 'vnet1',
'speed': '1000', 'state': 'unknown', 'rxErrors':
'0', 'rxDropped': '0', 'txErrors': '0',
'txDropped': '0', 'rx': '94068359', 'tx':
'107873946', 'sampleTime': 861814.22111595}}, 'disks':
{'sdc': {'truesize': '0', 'apparentsize': '0',
'readLatency': '0', 'writeLatency': '0',
'flushLatency': '0', 'writtenBytes': '0',
'writeOps': '0', 'readOps': '22', 'readBytes':
'406', 'readRate': '0.0', 'writeRate': '0.0'},
'vda': {'truesize': '76436357
12', 'apparentsize': '53689188352', 'readLatency':
'0', 'writeLatency': '18963692.40625', 'flushLatency':
'132326.26666666666', 'writtenBytes': '1946506752',
'writeOps': '165176', 'readOps': '31194',
'readBytes': '859866624', 'readRate': '0.0',
'writeRate': '14743.1258832571', 'imageID':
'bcd042b1-0978-43bd-bf24-eb1f554cd520'}}, 'balloonInfo':
{'balloon_max': '13489152', 'balloon_min': '13489152',
'balloon_cur': '13489152', 'balloon_target': '13489152',
'ballooning_enabled': True}, 'vcpuCount': '4',
'memoryStats': {'mem_total': '12887304', 'mem_unused':
'7100564', 'mem_free': '7994896', 'swap_in': 0,
'swap_out': 0, 'majflt': 0, 'minflt': 196, 'pageflt':
196}, 'displayInfo': [{'type': 'vnc', 'port':
'5900', 'tlsPort': '', 'ipAddress': '10.0.1.47'},
{'type': 'spice', 'port': '5901', 'tlsPort':
'5902', 'ipAddress': '10.0.1.47'}], 'hash':
'-7465763413854713032', 'vmJobs': {}, 'vcpuQuota': '-1',
'vcpuPeriod': 100000, 'username': 'root', 'session':
'Unknown', 'memUsage': '40', 'guestCPUCo
unt': -1, 'appsList': ('kernel-4.18.0-486.el8.x86_64',
'qemu-guest-agent-6.2.0'), 'guestIPs': '', 'guestFQDN':
'manager-hosted-engine.com', 'netIfaces': [{'hw':
'00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6':
['::1'], 'name': 'lo'}, {'hw':
'00:16:3e:54:7f:23', 'inet': ['10.0.1.48'], 'inet6':
['fe80::216:3eff:fe54:7f23'], 'name': 'eth0'}],
'disksUsage': [{'path': '/', 'total':
'7505707008', 'used': '4872785920', 'fs': 'xfs'},
{'path': '/var', 'total': '21464350720', 'used':
'1688195072', 'fs': 'xfs'}, {'path': '/home',
'total': '1063256064', 'used': '41271296', 'fs':
'xfs'}, {'path': '/tmp', 'total': '2136997888',
'used': '48943104', 'fs': 'xfs'}, {'path':
'/var/log', 'total': '10726932480', 'used':
'132661248', 'fs': 'xfs'}, {'path':
'/var/log/audit', 'total': '1063256064', 'used':
'42233856', 'fs': 'xfs'}, {'path': '/boot',
'total': '1063256064', 'used': '352448512', 'fs':
'xfs'}], 'guestName': 'manager-hosted-engine.com',
'guestOs': '4.18.0-486.el8.x86_64', 'guestOsInfo': {'t
ype': 'linux', 'arch': 'x86_64', 'kernel':
'4.18.0-486.el8.x86_64', 'distribution': 'CentOS Stream',
'version': '8', 'codename': ''}, 'guestTimezone':
{'offset': 330, 'zone': 'IST'}}]} from=::1,57200,
vmId=9ea50595-33cc-4fde-9cd2-ac35c4e97a2c (api:54)
2023-04-20 12:23:55,230+0530 INFO (jsonrpc/3) [vdsm.api] START
repoStats(domains=['963b7fd2-c32f-400d-9abf-ade3d702cb4b']) from=::1,57200,
task_id=0279b915-ac22-46d2-afbc-8466144d22d8 (api:48)
2023-04-20 12:23:55,231+0530 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats
return={'963b7fd2-c32f-400d-9abf-ade3d702cb4b': {'code': 0,
'lastCheck': '5.1', 'delay': '0.000864457',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,57200, task_id=0279b915-ac22-46d2-afbc-8466144d22d8 (api:54)
2023-04-20 12:23:55,735+0530 ERROR (monitor/6ac48b4) [storage.Monitor] Error checking
domain 6ac48b40-a9dc-4475-b17e-247b6018abc1 (monitor:453)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 434, in
_checkDomainStatus
self.domain.selftest()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 48, in
__getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 146, in
findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 136, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist:
('6ac48b40-a9dc-4475-b17e-247b6018abc1',)
2023-04-20 12:23:56,307+0530 ERROR (check/loop) [storage.Monitor] Error checking path
/rhev/data-center/mnt/[abcd:abcd:abcd::51]:6789:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9/6ac48b40-a9dc-4475-b17e-247b6018abc1/dom_md/metadata
(monitor:511)
====================engine.log================================
2023-04-20 12:23:55,064+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Lock Acquired to object
'EngineLock:{exclusiveLocks='[ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9=STORAGE_CONNECTION]',
sharedLocks=''}'
2023-04-20 12:23:55,118+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Running command:
AddStorageServerConnectionCommand internal: false. Entities affected : ID:
aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with
role type ADMIN
2023-04-20 12:23:55,119+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] START,
ConnectStorageServerVDSCommand(HostName = deployment-host,
StorageServerConnectionManagementVDSParameters:{hostId='745b7584-0a43-47d9-985f-af0a0155e787',
storagePoolId='00000000-0000-0000-0000-000000000000',
storageType='POSIXFS',
connectionList='[StorageServerConnections:{id='null',
connection='ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9',
iqn='null', vfsType='ceph',
mountOptions='rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
iface='null', netIfaceName='null'}]',
sendNetworkEventOnFailure='true'}), log id: 7e34045e
2023-04-20 12:23:55,223+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] FINISH, ConnectStorageServerVDSCommand,
return: {00000000-0000-0000-0000-000000000000=100}, log id: 7e34045e
2023-04-20 12:23:55,248+05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-37)
[31e859ef-97d0-435c-81ba-9d29bf637527] EVENT_ID: STORAGE_DOMAIN_ERROR(996), The error
message for connection
ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 returned
by VDSM was: General Exception
2023-04-20 12:23:55,248+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.FileStorageHelper] (default task-37)
[31e859ef-97d0-435c-81ba-9d29bf637527] The connection with details
'ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9'
failed because of error code '100' and error message is: general exception
2023-04-20 12:23:55,249+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'
failed: EngineException: GeneralException (Failed with error GeneralException and code
100)
2023-04-20 12:23:55,255+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Transaction rolled-back for command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'.
2023-04-20 12:23:55,278+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Lock freed to object
'EngineLock:{exclusiveLocks='[ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9=STORAGE_CONNECTION]',
sharedLocks=''}'
This also doesn't work.
We tried mounting the ceph-mon nodes cluster manually on the deployment host CLI , we were
able to mount the same.
sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
/rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 -o
rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==
[root@deployment-host mnt]# df -kh
df: /run/user/0/gvfs: Transport endpoint is not connected
Filesystem
Size Used Avail Use% Mounted on
[abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
19G 0 19G 0%
/rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9
Queries:
1. Could you help us in creating a storage domain for the same from UI.
2. Do we have any process of doing the same using the CLI.
Thanks and Regards
Kushagra Gupta