Hi Matthew,
Actually, your description is related to 2 features available for ovirt 4.4.5
1. The ability to switch the master storage domain while domains are up and running [1]
2. Clearing the finished tasks from REST API [2] and UI [3].

We recommend you upgrade your engine to enjoy those features.

In the meanwhile, as you've described, moving the Master role from one storage to the other is available using putting the domain into maintenance.
In order to clear the finished tasks from SPM:
   vdsm-client Host getAllTasksStatuses

It should be something like that:
{
    "1dc4d885-577a-4b6a-b01f-e682602a907c": {
        "code": 0,
        "message": "1 jobs completed successfully",
        "taskID": "1dc4d885-577a-4b6a-b01f-e682602a907c",
        "taskResult": "success",
        "taskState": "finished"
    }
}

Then clear that tasks:
   vdsm-client Task clear taskID=12345
Once it gets cleared, the reconstruction can be finished.

To verify there are no more finished async tasks, you can run this SQL query on the engine:
engine=# select * from async_tasks WHERE storage_pool_id = '123'; 

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1910022
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1627997
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1910302

Regards,
Shani Leviim


On Thu, Jul 29, 2021 at 8:33 AM Matthew Benstead <matthewb@uvic.ca> wrote:
Hello,

I'm trying to decommission the old master storage domain in ovirt, and replace it with a new one. All of the VMs have been migrated off of the old master, and everything has been running on the new storage domain for a couple months. But when I try to put the old domain into maintenance mode I get an error.

Old Master: vm-storage-ssd
New Domain: vm-storage-ssd2

The error is:

Failed to Reconstruct Master Domain for Data Center EDC2

As well as:

Sync Error on Master Domain between Host daccs01 and oVirt Engine. Domain: vm-storage-ssd is marked as Master in oVirt Engine database but not on the Storage side. Please consult with Support on how to fix this issue.

2021-07-28 11:41:34,870-07 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-23) [] Master domain version is not in sync between DB and VDSM. Domain vm-storage-ssd
 marked as master, but the version in DB: 283 and in VDSM: 280

And:

Not stopping SPM on vds daccs01, pool id f72ec125-69a1-4c1b-a5e1-313fcb70b6ff as there are uncleared tasks Task '5fa9edf0-56c3-40e4-9327-47bf7764d28d', status 'finished'


After a couple minutes all the domains are marked as active again and things continue, but vm-storage-ssd is still listed as the master domain. Any thoughts?

This is on 4.3.10.4-1.el7   on CentOS 7.

engine=# SELECT storage_name, storage_pool_id, storage, status FROM storage_pool_with_storage_domain ORDER BY storage_name;
     storage_name      |           storage_pool_id            |                storage                 | status
-----------------------+--------------------------------------+----------------------------------------+--------
 compute1-iscsi-ssd    | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | yvUESE-yWUv-VIWL-qX90-aAq7-gK0I-EqppRL |      1
 compute7-iscsi-ssd    | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | 8ekHdv-u0RJ-B0FO-LUUK-wDWs-iaxb-sh3W3J |      1
 export-domain-storage | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | d3932528-6844-481a-bfed-542872ace9e5   |      1
 iso-storage           | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | f800b7a6-6a0c-4560-8476-2f294412d87d   |      1
 vm-storage-7200rpm    | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | a0bff472-1348-4302-a5c7-f1177efa45a9   |      1
 vm-storage-ssd        | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | 95acd9a4-a6fb-4208-80dd-1c53d6aacad0   |      1
 vm-storage-ssd2       | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | 829d0600-c3f7-4dae-a749-d7f05c6a6ca4   |      1
(7 rows)

Thanks,
 -Matthew
--
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OXOXW6B2NWXOUGZV3OKO4OMDXVDJSQLZ/