11:18 a.m.
(I'm always talking about the last ovirt version that is currently engine:
4.5.2.5-1.el8, node: ovirt-node-ng-4.5.2-0.20220810.0)
I tried to add an iSCSI storage domain adding a new vlan network where the storage is
attached.... this gave a completely mess-up of the storage engine.
I would like to restart the procedure again and to accomplish this I have do remove the
storage domain.
To do this, I added manually the vlan to the nodes as I'm configured using linux
bridge simple networking, so I discovered I have not to manage ovs environment.
I currently have the storage domain in maintenance mode and I've successfully
regenerated the discovery informations as the network to connecto to the storage is
online.
But I cannot detach the storage and I cannot activate the storage domain, so I'm
completely stuck.
This is the related log in the engine while I was trying to detach the storage:
2022-09-26 07:20:48,043Z ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-55008) [7a5a5413-36d0-4a15-b555-c9801f92483d]
Command 'DetachStorageDomainVDSCommand(
DetachStorageDomainVDSCommandParameters:{storagePoolId='da146814-f823-40e0-bd7b-8478dcfa38cd',
ignoreFailoverLimit='false',
storageDomainId='aac79175-ab2b-4b5b-a6e4-9feef9ce17ab',
masterDomainId='00000000-0000-0000-0000-000000000000', masterVersion='8',
force='false'})' execution failed: IRSGenericException: IRSErrorException:
Storage domain not in pool: 'domain=aac79175-ab2b-4b5b-a6e4-9feef9ce17ab,
pool=da146814-f823-40e0-bd7b-8478dcfa38cd'
as you can see it seem there is a problem in the engine database.
I saw here (https://lists.ovirt.org/pipermail/users/2015-January/030504.html) that someone
worked directly on the engine db to delete the storage domain but I read somewhere else
that the domain information is also mapped in a lvm volume metadata. What can I do from
here?
Thank you.
7:22 p.m.
New subject: Storage domain deletion not possible because not possible to detach
I tried to do what written in the list.
This is what I did:
[root@ovirt-node3 ~]# hosted-engine --set-maintenance --mode=global
[root@ovirt-engine ~]# systemctl list-units | grep ovirt
ovirt-engine-dwhd.service
loaded active running oVirt Engine Data Warehouse
ovirt-engine.service
loaded active running oVirt Engine
ovirt-fence-kdump-listener.service
loaded active running oVirt Engine fence_kdump listener
ovirt-imageio.service
loaded active running oVirt ImageIO Daemon
ovirt-provider-ovn.service
loaded active running oVirt OVN provider
ovirt-vmconsole-proxy-sshd.service
loaded active running oVirt VM Console SSH server daemon
ovirt-websocket-proxy.service
loaded active running oVirt Engine websockets proxy
dev-mapper-ovirt\x2dswap.swap
loaded active active /dev/mapper/ovirt-swap
[root@ovirt-engine ~]# systemctl stop ovirt-*
[root@ovirt-engine ~]# systemctl list-units | grep ovirt
dev-mapper-ovirt\x2dswap.swap
loaded active active /dev/mapper/ovirt-swap
[root@ovirt-engine ~]# PGPASSWORD="Gu8xDOW8a0Fvfqg26bsyqf" psql --user=engine -h
127.0.0.1 engine
psql (12.11)
Type "help" for help.
engine=# select id, storage_name from storage_domain_static;
id | storage_name
--------------------------------------+------------------------
072fbaa1-08f3-4a40-9f34-a5ca22dd1d74 | ovirt-image-repository
493cc639-8207-4478-a07e-a79d9765c8d1 | ISO_at_node2
d79605bc-f02a-47d1-b35a-b7a3ff2907e3 | gv0
45b4f14c-8323-482f-90ab-99d8fd610018 | gv1
3577c21e-f757-4405-97d1-0f827c9b4e22 | hosted_storage
fdbed4cd-400b-4db1-8f99-e400787445d9 | domainATnetworkstorage
aac79175-ab2b-4b5b-a6e4-9feef9ce17ab | 7000a28
(7 rows)
engine=# select storage_domain_id, ovf_disk_id from storage_domains_ovf_info where
storage_domain_id = 'aac79175-ab2b-4b5b-a6e4-9feef9ce17ab';
storage_domain_id | ovf_disk_id
--------------------------------------+--------------------------------------
aac79175-ab2b-4b5b-a6e4-9feef9ce17ab | e150cef7-91d6-4538-b6a7-d185a1b02509
aac79175-ab2b-4b5b-a6e4-9feef9ce17ab | 491920f8-a06e-4702-aa17-8af4589374b4
(2 rows)
engine=# delete from storage_domain_dynamic where id =
'aac79175-ab2b-4b5b-a6e4-9feef9ce17ab';
DELETE 1
engine=# delete from storage_domain_static where id =
'aac79175-ab2b-4b5b-a6e4-9feef9ce17ab';
DELETE 1
engine=# delete from base_disks where disk_id in
('e150cef7-91d6-4538-b6a7-d185a1b02509','491920f8-a06e-4702-aa17-8af4589374b4');
DELETE 2
engine=# delete from storage_domains_ovf_info where storage_domain_id =
'aac79175-ab2b-4b5b-a6e4-9feef9ce17ab';
DELETE 0
engine=# delete from storage_pool_iso_map where storage_id =
'aac79175-ab2b-4b5b-a6e4-9feef9ce17ab';
DELETE 0
[root@ovirt-node2 ~]# lvmdevices
Device /dev/mapper/Samsung_SSD_870_EVO_4TB_S6BCNG0R300064E IDTYPE=mpath_uuid
IDNAME=mpath-Samsung_SSD_870_EVO_4TB_S6BCNG0R300064E
DEVNAME=/dev/mapper/Samsung_SSD_870_EVO_4TB_S6BCNG0R300064E
PVID=1ScYNcZ0GcuR7j9xYezuh562K2GX4f7h
Device /dev/mapper/ST4000NM000A-2HZ100_WJG1ZC85 IDTYPE=mpath_uuid
IDNAME=mpath-ST4000NM000A-2HZ100_WJG1ZC85 DEVNAME=/dev/mapper/ST4000NM000A-2HZ100_WJG1ZC85
PVID=PALZBWIFpZSGNlZo1uamMhDdauYrsXb8
Device /dev/nvme0n1p3 IDTYPE=sys_wwid IDNAME=eui.6479a74c40200874 DEVNAME=/dev/nvme0n1p3
PVID=S22fRYq8H37g23RVdBshQLotkqsHwYz2 PART=3
Device /dev/nvme0n1p4 IDTYPE=sys_wwid IDNAME=eui.6479a74c40200874 DEVNAME=/dev/nvme0n1p4
PVID=9vfdUZXH9puCBiBf7BNLK0bI3Q485xdx PART=4
--> Device /dev/mapper/3600507640082010b380000000000002a IDTYPE=mpath_uuid
IDNAME=mpath-3600507640082010b380000000000002a
DEVNAME=/dev/mapper/3600507640082010b380000000000002a
PVID=uVTp8Un0vqElbC4SdayZ0L8o5jIK0Hdi
engine=# select * from luns;
physical_volume_id | lun_id |
volume_group_id | serial | lun_mapping | vendor_id |
product_id | device_size | discard_max_size
----------------------------------------+-----------------------------------+----------------------------------------+----------------------------+-------------+-----------+------------+-------------+------------------
uVTp8U-n0vq-ElbC-4Sda-yZ0L-8o5j-IK0Hdi | 3600507640082010b380000000000002a |
lIX0VF-h1xs-u4m5-BYiO-8iwB-Gjxh-xWkQWr | SIBM_2145_0100208042ceXX00 | 0 | IBM
| 2145 | 1024 | 2097152
(1 row)
engine=# delete from luns where physical_volume_id =
'uVTp8U-n0vq-ElbC-4Sda-yZ0L-8o5j-IK0Hdi';
DELETE 1
engine=# select * from storage_server_connections;
id | connection
| user_name | password | iqn | port | portal |
storage_type | mo
unt_options | vfs_type | nfs_version | nfs_timeo | nfs_retrans |
gluster_volume_id
--------------------------------------+--------------------------------------------------------+-----------+----------+-------------------------------------------+------+--------+--------------+-------------------------
-----------------------------------+-----------+-------------+-----------+-------------+--------------------------------------
4ecbbb11-7608-4244-8aad-273110dbca91 | localhost:/glen
| | | | | 1 |
7 |
| glusterfs | | | |
db369b1e-6065-4a2b-848c-0e16ea5dfc6d | ovirt-node2.ovirt:/dati/vd0/nfs_at_node2
| | | | | 1 |
1 |
| | auto | | |
dcd8001a-5ac8-4bd3-aa7c-3a300c6e6971 | ovirt-node2.ovirt:/gv1
| | | | | 1 |
7 | backup-volfile-servers=o
virt-node3.ovirt:ovirt-node4.ovirt | glusterfs | | | |
863221f4-e11c-4589-95e9-aa3948e177f5
402e4f09-3c47-4b2b-a5fd-67957a5a698e | ovirt-node2.ovirt:/gv0
| | | | | 1 |
7 | backup-volfile-servers=o
virt-node3.ovirt:ovirt-node4.ovirt | glusterfs | | | |
187320c6-2d12-409f-91b0-d22179903eaf
-> 09e9aa89-7e92-4505-bff8-75298862de89 | 169.254.30.26
| | | iqn.1986-03.com.ibm:2145.gav7000a28.node1 | 3260 | 4
| 3 |
| | | | |
aedbec57-fcfc-419e-b7f5-253fc7bc4083 |
networkstorage.ovirt:/dati/networkstorage/ovirt_domain | | |
| | 1 | 1 |
| | auto | | |
(6 rows)
engine=# delete from storage_server_connections where
id='09e9aa89-7e92-4505-bff8-75298862de89';
DELETE 1
-- I don't have any disk mapped in the storage domain
engine=# -- not done delete from base_disks where
disk_id='03d651eb-14a9-4dca-8c87-605f101a5e0c';
engine=# -- not done delete from permissions where
object_id='03d651eb-14a9-4dca-8c87-605f101a5e0c';
***********************
Searching for objects:
*** Storage uuid: aac79175-ab2b-4b5b-a6e4-9feef9ce17ab
engine=# select * from job_subject_entity where
entity_id='aac79175-ab2b-4b5b-a6e4-9feef9ce17ab';
job_id | entity_id | entity_type
--------------------------------------+--------------------------------------+-------------
34a3912a-970f-4262-94a8-9ef25bf2a0c6 | aac79175-ab2b-4b5b-a6e4-9feef9ce17ab | Storage
c6053af1-21d8-4917-85e5-b63176a15708 | aac79175-ab2b-4b5b-a6e4-9feef9ce17ab | Storage
(2 rows)
*** ovf_disk_id e150cef7-91d6-4538-b6a7-d185a1b02509 491920f8-a06e-4702-aa17-8af4589374b4
engine=# select * from images where image_group_id in
('e150cef7-91d6-4538-b6a7-d185a1b02509','491920f8-a06e-4702-aa17-8af4589374b4');
image_guid | creation_date | size |
it_guid | parentid | imagestatus |
lastmodified | vm_snapshot_id | volu
me_type | volume_format | image_group_id | _create_date
| _update_date | active | volume_classification | qcow_compat |
sequence_number
--------------------------------------+------------------------+-----------+--------------------------------------+--------------------------------------+-------------+----------------------------+----------------+-----
--------+---------------+--------------------------------------+-------------------------------+-------------------------------+--------+-----------------------+-------------+-----------------
b9bf1dc5-358d-4aa0-b8ee-d00781f9a40d | 2022-09-21 15:06:00+00 | 134217728 |
00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 | 1
| 2022-09-21 16:10:51.461+00 | |
1 | 5 | e150cef7-91d6-4538-b6a7-d185a1b02509 | 2022-09-21
18:13:11.824767+00 | 2022-09-21 16:10:52.430083+00 | t | 0 |
0 | 1
a36e1483-bad5-4737-8efd-b088d8488a3a | 2022-09-21 15:06:00+00 | 134217728 |
00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 | 1
| 2022-09-21 16:10:51.461+00 | |
1 | 5 | 491920f8-a06e-4702-aa17-8af4589374b4 | 2022-09-21
18:13:09.806755+00 | 2022-09-21 16:10:53.265084+00 | t | 0 |
0 | 1
(2 rows)
[root@ovirt-engine ~]# halt -p
[root@ovirt-node3 ~]# hosted-engine --set-maintenance --mode=none
wait... and then the domain disappeared and the engine seems to work.
I think there must be some other housekeeping as I have at least some garbage in the
job_subject_entity, and images tables.
Hope this helps someone
783
days inactive
785
days old
1 comments
1 participants
participants (1)
-
Diego Ercolani