Hi team, We have installed ovirt 4.4. We have a self hosted Engine setup in the environment which has 1 hosted engine on top of 1 deployment-host. Goal: We want to create a storage domain of POSIX compliant type for mounting a ceph based infrastructure. We have done SRV based resolution in our DNS server but we are unable to create a storage domain. Issue: We are passing the following information: path: :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 vfs-type: ceph mounting option:rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA= We get the following errors: ====================vdsm.log================================== 2023-04-20 11:26:30,318+0530 INFO (jsonrpc/7) [storage.Mount] mounting :/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 at /rhev/data-center/mnt/ :_volumes_xyz_conf_2ee9c2d0-873b-4d04-8c46-4c0da02787b8 (mount:207) 2023-04-20 11:26:30,384+0530 ERROR (jsonrpc/7) [storage.HSM] Could not connect to storageServer (hsm:2374) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2371, in connectStorageServer conObj.connect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 180, in connect six.reraise(t, v, tb) File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 171, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 210, in mount cgroup=cgroup) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File "<string>", line 2, in mount File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'ceph', '-o', 'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', ' :/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', '/rhev/data-center/mnt/ :_volumes_xyz_conf_2ee9c2d0-873b-4d04-8c46-4c0da02787b8'] failed with rc=32 out=b'' err=b'mount error 3 = No such process\n' 2023-04-20 11:31:05,715+0530 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH connectStorageServer error=:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 is not a valid hosttail address: (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/common/network/address.py", line 42, in hosttail_split raise ValueError ValueError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 110, in wrapper return m(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1190, in prepare raise self.error File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in _run return fn(*args, **kargs) File "<decorator-gen-117>", line 2, in connectStorageServer File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2368, in connectStorageServer conObj = storageServer.ConnectionFactory.createConnection(conInfo) File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 741, in createConnection return ctor(**params) File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 154, in __init__ self._remotePath = fileUtils.normalize_path(spec) File "/usr/lib/python3.6/site-packages/vdsm/storage/fileUtils.py", line 98, in normalize_path host, tail = address.hosttail_split(path) File "/usr/lib/python3.6/site-packages/vdsm/common/network/address.py", line 45, in hosttail_split raise HosttailError('%s is not a valid hosttail address:' % hosttail) vdsm.common.network.address.HosttailError: :/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 is not a valid hosttail address: 2023-04-20 11:31:05,715+0530 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer failed (error 451) in 0.00 seconds (__init__:312) ====================engine.log================================ 2023-04-20 11:31:03,818+05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] HostName = deployment-host 2023-04-20 11:31:03,818+05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Command 'ConnectStorageServerVDSCommand(HostName = deployment-host, StorageServerConnectionManagementVDSParameters:{hostId='745b7584-0a43-47d9-985f-af0a0155e787', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='POSIXFS', connectionList='[StorageServerConnections:{id='null', connection=':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', iqn='null', vfsType='ceph', mountOptions='rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'})' execution failed: VDSGenericException: VDSErrorException: Failed to ConnectStorageServerVDS, error = Error storage server connection: ("domType=6, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'password': '********', 'vfs_type' : 'ceph', 'port': '', 'mnt_options': 'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', 'iqn': '', 'connection': ':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', 'ipv6_enabled': 'false', 'id': '00000000-0000-0000-0000-000000000000', 'user': '', 'tpgt': '1'}]",), code = 451 2023-04-20 11:31:03,818+05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] FINISH, ConnectStorageServerVDSCommand, return: , log id: 19c06f3f 2023-04-20 11:31:03,819+05 ERROR [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Command 'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ConnectStorageServerVDS, error = Error storage server connection: ("domType=6, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'password': '********', 'vfs_type': 'ceph', 'port': '', 'mnt_options': 'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', 'iqn': '', 'connection': ':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8', 'ipv6_enabled': 'false', 'id': '00000000-0000-0000-0000-000000000000', 'user': '', 'tpgt': '1'}]",), code = 451 (Failed with error StorageServerConnectionError and code 451) 2023-04-20 11:31:03,824+05 ERROR [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'. 2023-04-20 11:31:03,847+05 INFO [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Lock freed to object 'EngineLock:{exclusiveLocks='[:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8=STORAGE_CONNECTION]', sharedLocks=''}' 2023-04-20 11:34:27,788+05 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [63fbb9de] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c6e0aa38-20ff-4be3-9b34-81a7b0fabb6a=PROVIDER]', sharedLocks=''}' 2023-04-20 11:34:27,809+05 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [63fbb9de] Running command: SyncNetworkProviderCommand internal: true. 2023-04-20 11:34:28,017+05 INFO [org.ovirt.engine.core.sso.service.AuthenticationService] (default task-29) [] User admin@internal-authz with profile [internal] successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access =============================================================================== We tried creating a NFS type storage domain using the hostname of the nfs server. This hostname is resolved using the dns server. We are able to create this storage domain. We also tried creating the POSIX compliant domain using the hostname of the ceph-mon node but it is failing. We are passing the following information: path: ceph-node2:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 vfs-type: ceph mounting option:rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA= In this scenario though the mounting is happening but the storage domain is not created correctly. we are getting the following errors: ====================vdsm.log================================== 2023-04-20 12:23:54,759+0530 INFO (jsonrpc/0) [api.virt] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': [{'statusTime': '3009297867', 'status': 'Up', 'vmId': '9ea50595-33cc-4fde-9cd2-ac35c4e97a2c', 'vmName': 'HostedEngine', 'vmType': 'kvm', 'kvmEnable': 'true', 'acpiEnable': 'true', 'elapsedTime': '71325', 'monitorResponse': '0', 'clientIp': '', 'timeOffset': '0', 'pauseCode': 'NOERR', 'cpuUser': '7.12', 'cpuSys': '1.80', 'cpuUsage': '1264730000000', 'network': {'vnet1': {'macAddr': '00:16:3e:54:7f:23', 'name': 'vnet1', 'speed': '1000', 'state': 'unknown', 'rxErrors': '0', 'rxDropped': '0', 'txErrors': '0', 'txDropped': '0', 'rx': '94068359', 'tx': '107873946', 'sampleTime': 861814.22111595}}, 'disks': {'sdc': {'truesize': '0', 'apparentsize': '0', 'readLatency': '0', 'writeLatency': '0', 'flushLatency': '0', 'writtenBytes': '0', 'writeOps': '0', 'readOps': '22', 'readBytes': '406', 'readRate': '0.0', 'writeRate': '0.0'}, 'vda': {'truesize': '76436357 12', 'apparentsize': '53689188352', 'readLatency': '0', 'writeLatency': '18963692.40625', 'flushLatency': '132326.26666666666', 'writtenBytes': '1946506752', 'writeOps': '165176', 'readOps': '31194', 'readBytes': '859866624', 'readRate': '0.0', 'writeRate': '14743.1258832571', 'imageID': 'bcd042b1-0978-43bd-bf24-eb1f554cd520'}}, 'balloonInfo': {'balloon_max': '13489152', 'balloon_min': '13489152', 'balloon_cur': '13489152', 'balloon_target': '13489152', 'ballooning_enabled': True}, 'vcpuCount': '4', 'memoryStats': {'mem_total': '12887304', 'mem_unused': '7100564', 'mem_free': '7994896', 'swap_in': 0, 'swap_out': 0, 'majflt': 0, 'minflt': 196, 'pageflt': 196}, 'displayInfo': [{'type': 'vnc', 'port': '5900', 'tlsPort': '', 'ipAddress': '10.0.1.47'}, {'type': 'spice', 'port': '5901', 'tlsPort': '5902', 'ipAddress': '10.0.1.47'}], 'hash': '-7465763413854713032', 'vmJobs': {}, 'vcpuQuota': '-1', 'vcpuPeriod': 100000, 'username': 'root', 'session': 'Unknown', 'memUsage': '40', 'guestCPUCo unt': -1, 'appsList': ('kernel-4.18.0-486.el8.x86_64', 'qemu-guest-agent-6.2.0'), 'guestIPs': '', 'guestFQDN': 'manager-hosted-engine.com', 'netIfaces': [{'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6': ['::1'], 'name': 'lo'}, {'hw': '00:16:3e:54:7f:23', 'inet': ['10.0.1.48'], 'inet6': ['fe80::216:3eff:fe54:7f23'], 'name': 'eth0'}], 'disksUsage': [{'path': '/', 'total': '7505707008', 'used': '4872785920', 'fs': 'xfs'}, {'path': '/var', 'total': '21464350720', 'used': '1688195072', 'fs': 'xfs'}, {'path': '/home', 'total': '1063256064', 'used': '41271296', 'fs': 'xfs'}, {'path': '/tmp', 'total': '2136997888', 'used': '48943104', 'fs': 'xfs'}, {'path': '/var/log', 'total': '10726932480', 'used': '132661248', 'fs': 'xfs'}, {'path': '/var/log/audit', 'total': '1063256064', 'used': '42233856', 'fs': 'xfs'}, {'path': '/boot', 'total': '1063256064', 'used': '352448512', 'fs': 'xfs'}], 'guestName': 'manager-hosted-engine.com', 'guestOs': '4.18.0-486.el8.x86_64', 'guestOsInfo': {'t ype': 'linux', 'arch': 'x86_64', 'kernel': '4.18.0-486.el8.x86_64', 'distribution': 'CentOS Stream', 'version': '8', 'codename': ''}, 'guestTimezone': {'offset': 330, 'zone': 'IST'}}]} from=::1,57200, vmId=9ea50595-33cc-4fde-9cd2-ac35c4e97a2c (api:54) 2023-04-20 12:23:55,230+0530 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=['963b7fd2-c32f-400d-9abf-ade3d702cb4b']) from=::1,57200, task_id=0279b915-ac22-46d2-afbc-8466144d22d8 (api:48) 2023-04-20 12:23:55,231+0530 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'963b7fd2-c32f-400d-9abf-ade3d702cb4b': {'code': 0, 'lastCheck': '5.1', 'delay': '0.000864457', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,57200, task_id=0279b915-ac22-46d2-afbc-8466144d22d8 (api:54) 2023-04-20 12:23:55,735+0530 ERROR (monitor/6ac48b4) [storage.Monitor] Error checking domain 6ac48b40-a9dc-4475-b17e-247b6018abc1 (monitor:453) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 434, in _checkDomainStatus self.domain.selftest() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 48, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 146, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 136, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('6ac48b40-a9dc-4475-b17e-247b6018abc1',) 2023-04-20 12:23:56,307+0530 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/[abcd:abcd:abcd::51]:6789:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9/6ac48b40-a9dc-4475-b17e-247b6018abc1/dom_md/metadata (monitor:511) ====================engine.log================================ 2023-04-20 12:23:55,064+05 INFO [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9=STORAGE_CONNECTION]', sharedLocks=''}' 2023-04-20 12:23:55,118+05 INFO [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2023-04-20 12:23:55,119+05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] START, ConnectStorageServerVDSCommand(HostName = deployment-host, StorageServerConnectionManagementVDSParameters:{hostId='745b7584-0a43-47d9-985f-af0a0155e787', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='POSIXFS', connectionList='[StorageServerConnections:{id='null', connection='ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9', iqn='null', vfsType='ceph', mountOptions='rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 7e34045e 2023-04-20 12:23:55,223+05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 7e34045e 2023-04-20 12:23:55,248+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] EVENT_ID: STORAGE_DOMAIN_ERROR(996), The error message for connection ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 returned by VDSM was: General Exception 2023-04-20 12:23:55,248+05 ERROR [org.ovirt.engine.core.bll.storage.connection.FileStorageHelper] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] The connection with details 'ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9' failed because of error code '100' and error message is: general exception 2023-04-20 12:23:55,249+05 ERROR [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Command 'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand' failed: EngineException: GeneralException (Failed with error GeneralException and code 100) 2023-04-20 12:23:55,255+05 ERROR [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'. 2023-04-20 12:23:55,278+05 INFO [org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand] (default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Lock freed to object 'EngineLock:{exclusiveLocks='[ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9=STORAGE_CONNECTION]', sharedLocks=''}' This also doesn't work. We tried mounting the ceph-mon nodes cluster manually on the deployment host CLI , we were able to mount the same. sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 -o rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA== [root@deployment-host mnt]# df -kh df: /run/user/0/gvfs: Transport endpoint is not connected Filesystem Size Used Avail Use% Mounted on [abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 19G 0 19G 0% /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 Queries: 1. Could you help us in creating a storage domain for the same from UI. 2. Do we have any process of doing the same using the CLI. Thanks and Regards Kushagra Gupta