FC Storage recover
by csabany＠freemail.hu
Hi,
We have a 4 node cluster (4.4.10) and a standalone manager.
We attached to this cluster 4 volumes on FC.
Last week we lost the storage canisters (both), but recovered one of them from the storage disks.
Unfortunatly only 3 storages came back from offline.
I see the 4th FC ID, but it's on "Can't Extend" status.
If i check the storages on nodes (lsblk), i can see only the ID not more on this 4th storage.
Do you know any recovery tool or any idea for this situation?
Thanks
csabany
2 years, 8 months
OVA Export from Snapshot?
by Alan G
Hi,
Is there a way to export an OVA from a snapshot? Or only from the "Active VM" image?
Thanks,
Alan
2 years, 8 months
CLI Offline Engine Deployment failing to activate storage domain to gluster error 400
by Abe E
Due to some complications in my 4.4 to 4.5 upgrade that have taken down ovirt for a few days I must go back to 4.4 to salvage what I can and rethink how I will upgrade.
Due to that im having to rebuild my engine although I took backups prior.
My issue currently is with 4.4 the engine deployment has an error regarding mirrors so I was able to use the ansible script to deploy offline and I am at the activate storage domain portion where it fails
I am setting it to glusterfs with the host:/path as gluster-1.d.com:/engine
The host machine i am building the engine on has a mount:
gluster-1.d.com:/engine 104804356 13827696 90976660 14% /rhev/data-center/mnt/glusterSD/gluster-1.d.com:_engine
within the mapping :
drwxr-xr-x. 2 vdsm kvm 6 Apr 5 22:15 gluster-1.d.com:_data
drwxr-xr-x. 4 vdsm kvm 38 May 5 15:00 gluster-1.d.com:_engine
I am able to get into the temp engines ssh and ping the domain although I cannot seem to find where my steps are incorrect.
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain]
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."}
Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]:
VDSM Logs are showing:
2022-05-05 15:30:32,719-0600 INFO (jsonrpc/6) [storage.StorageDomain] sdUUID=7f7cf7a4-f8fa-41a4-8e6a-275230655a8c (fileSD:535)
2022-05-05 15:30:32,734-0600 INFO (jsonrpc/6) [vdsm.api] FINISH getStorageDomainInfo error=Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',) from=::ffff:192.168.222.116,53112, flow_id=661d6831, task_id=480acb07-0a53-4bad-aa30-4cd18ce9e5f2 (api:52)
2022-05-05 15:30:32,734-0600 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='480acb07-0a53-4bad-aa30-4cd18ce9e5f2') Unexpected error (task:877)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in _run
return fn(*args, **kargs)
File "<decorator-gen-131>", line 2, in getStorageDomainInfo
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2717, in getStorageDomainInfo
dom = self.validateSdUUID(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 313, in validateSdUUID
sdDom.validate()
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 538, in validate
raise se.StorageDomainAccessError(self.sdUUID)
vdsm.storage.exception.StorageDomainAccessError: Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',)
2022-05-05 15:30:32,734-0600 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='480acb07-0a53-4bad-aa30-4cd18ce9e5f2') aborting: Task is aborted: "value=Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',) abortedcode=379" (task:1182)
2022-05-05 15:30:32,735-0600 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getStorageDomainInfo error=Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',) (dispatcher:83)
2022-05-05 15:30:32,735-0600 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 379) in 0.02 seconds (__init__:312)
2022-05-05 15:30:32,743-0600 INFO (jsonrpc/0) [vdsm.api] START createStoragePool(spUUID='fd385045-8355-4ef8-9970-ade84698ad89', poolName='LabNet', masterDom='7f7cf7a4-f8fa-41a4-8e6a-275230655a8c', domList=['7f7cf7a4-f8fa-41a4-8e6a-275230655a8c'], masterVersion=13, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3) from=::ffff:192.168.222.116,53112, flow_id=661d6831, task_id=b58a0503-98c3-41a9-b5d7-3330e2985133 (api:48)
2022-05-05 15:30:32,743-0600 INFO (jsonrpc/0) [storage.StoragePool] updating pool fd385045-8355-4ef8-9970-ade84698ad89 backend from type NoneType instance 0x7f767080b9d0 to type StoragePoolDiskBackend instance 0x7f760c4dd308 (sp:168)
2022-05-05 15:30:32,743-0600 INFO (jsonrpc/0) [storage.StoragePool] spUUID=fd385045-8355-4ef8-9970-ade84698ad89 poolName=LabNet master_sd=7f7cf7a4-f8fa-41a4-8e6a-275230655a8c domList=['7f7cf7a4-f8fa-41a4-8e6a-275230655a8c'] masterVersion=13 {'LEASERETRIES': 3, 'LEASETIMESEC': 60, 'LOCKRENEWALINTERVALSEC': 5, 'IOOPTIMEOUTSEC': 10} (sp:624)
2022-05-05 15:30:32,743-0600 INFO (jsonrpc/0) [storage.StorageDomain] sdUUID=7f7cf7a4-f8fa-41a4-8e6a-275230655a8c (fileSD:535)
2022-05-05 15:30:32,759-0600 ERROR (jsonrpc/0) [storage.StoragePool] Unexpected error (sp:640)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 635, in create
domain.validate()
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 538, in validate
raise se.StorageDomainAccessError(self.sdUUID)
vdsm.storage.exception.StorageDomainAccessError: Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',)
2022-05-05 15:30:32,759-0600 INFO (jsonrpc/0) [vdsm.api] FINISH createStoragePool error=Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',) from=::ffff:192.168.222.116,53112, flow_id=661d6831, task_id=b58a0503-98c3-41a9-b5d7-3330e2985133 (api:52)
2022-05-05 15:30:32,759-0600 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='b58a0503-98c3-41a9-b5d7-3330e2985133') Unexpected error (task:877)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 635, in create
domain.validate()
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 538, in validate
raise se.StorageDomainAccessError(self.sdUUID)
vdsm.storage.exception.StorageDomainAccessError: Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in _run
return fn(*args, **kargs)
File "<decorator-gen-31>", line 2, in createStoragePool
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 1006, in createStoragePool
leaseParams)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 641, in create
raise se.StorageDomainAccessError(sdUUID)
vdsm.storage.exception.StorageDomainAccessError: Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',)
2022-05-05 15:30:32,759-0600 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='b58a0503-98c3-41a9-b5d7-3330e2985133') aborting: Task is aborted: "value=Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',) abortedcode=379" (task:1182)
2022-05-05 15:30:32,759-0600 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStoragePool error=Domain is either partially accessible or entirely inaccessible: ('7f7cf7a4-f8fa-41a4-8e6a-275230655a8c',) (dispatcher:83)
2022-05-05 15:30:32,759-0600 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StoragePool.create failed (error 379) in 0.02 seconds (__init__:312)
2022-05-05 15:30:34,709-0600 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=1b759c4f-ab60-4f3a-8ede-2606f916bf4a (api:48)
2022-05-05 15:30:34,710-0600 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task_id=1b759c4f-ab60-4f3a-8ede-2606f916bf4a (api:54)
2022-05-05 15:30:36,324-0600 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.222.116,53112 (api:48)
2022-05-05 15:30:36,344-0600 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.222.116,53112, task_id=3b40703b-83dc-4aff-ae92-41158e5ffefc (api:48)
2022-05-05 15:30:36,344-0600 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.222.116,53112, task_id=3b40703b-83dc-4aff-ae92-41158e5ffefc (api:54)
2022-05-05 15:30:36,344-0600 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::ffff:192.168.222.116,53112, task_id=00953608-b566-4d3d-be62-f20ea77ff7d9 (api:48)
2022-05-05 15:30:36,344-0600 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.222.116,53112, task_id=00953608-b566-4d3d-be62-f20ea77ff7d9 (api:54)
2022-05-05 15:30:36,348-0600 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.222.116,53112 (api:54)
2022-05-05 15:30:36,889-0600 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=a5db87b7-5130-49da-a5c4-5622dbe9d74b (api:48)
2022-05-05 15:30:36,889-0600 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=a5db87b7-5130-49da-a5c4-5622dbe9d74b (api:54)
2022-05-05 15:30:36,889-0600 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:735)
2022-05-05 15:30:37,444-0600 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,33062 (api:48)
2022-05-05 15:30:37,446-0600 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,33062 (api:54)
2022-05-05 15:30:37,451-0600 INFO (jsonrpc/3) [api.host] START getAllVmIoTunePolicies() from=::1,33062 (api:48)
2022-05-05 15:30:37,452-0600 INFO (jsonrpc/3) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'}, 'io_tune_policies_dict': {'da57699e-e76a-4125-81ce-50821bb09eef': {'policy': [], 'current_values': []}}} from=::1,33062 (api:54)
2 years, 8 months
vnc certificate renew
by csabany＠freemail.hu
Hi,
LAst month a renewed our hosts certificates by the "Enroll certificates" method.
The "/etc/pki/vdsm/libvirt-vnc/server-cert.pem" certificate wasn't renewed on my nodes (other certificates were).
How can i renew this certificate too?
thanks
csabany
2 years, 8 months
Backing up VMs with qemu-img
by David White
I've recently been working with the qemu-img commands for some work that has nothing to do with oVirt or anything inside an oVirt environment.
But learning and using these commands have given me an idea for automating backups.
I believe that the following is true, but to confirm, would the qemu-img commands be available on the oVirt hosts to take VM snapshots and disk images?
Sent with ProtonMail secure email.
2 years, 8 months
OVIRT Engine Restore from old Backup
by Abe E
Hey All, I really dont like spamming the forums but the issues I am having keep changing entirely.
I am at the point where I can get much closer to rebuilding my engine although my hurtle currently is restoring.
When I restore using my most recent backup it is problematic
Currently I am restoring by:
clean engine install
engine-cleanup
engine-backup --mode=restore --file=ovirt-engine-backup-20220504011012.backup --log=restore5.log
engine-setup
Comes up with alot of annoyances as the ID of the old hosted_storage and the new one are mismatched in the GUI.
On top of that it is not allowing me to deploy the engine to the 2nd Gluster (3rd one keeps deleting its glusters on reboot since 4.3 Upgrade).
Host 1 which runs the hosted engine is not showing as active or unlike, rather it is unresponsive, when I tried to enroll certificates it finally accepted but failed because it said its missing networks, once added it crashed the engine and ha agent wouldnt come up. Now that I think of it, I could have removed the extra networks from being required although it still doesnt benefit me here as I am unable to deploy or reinstall the 2nd Gluster node and the first one will not show its correct state as the host.
Should I just be deploying the DB, or is the full file backup fine? My issue here is it takes long to redeploy so the trial an error of failing takes me hours as my backup is from 4.5 engine even though my nodes are on 4.4 as 4.5 keeps causing the catastrophic failure of my whole cluster.
2 years, 8 months
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[XML error]". HTTP response code is 400.
by yp414＠163.com
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the storage interface to be up]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check local VM dir stat]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce local VM dir existence]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch host facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster version]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster major version]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster minor version]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set storage_format]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add NFS storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add glusterfs storage domain]
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[XML error]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[XML error]\". HTTP response code is 400."}
2 years, 8 months
Re: ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[XML error]". HTTP response code is 400.
by Jonas Liechti
I had the same problem and found the solution here: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6IKZ45B2TUC...
Find cli.py on the host and comment out the line with `value ['stripeCount'] = el.find('stripeCount').text`
Am 06.05.2022 19:01 schrieb yp414(a)163.com:
>
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the storage interface to be up]
> [ INFO ] skipping: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check local VM dir stat]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce local VM dir existence]
> [ INFO ] skipping: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch host facts]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster ID]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster facts]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter facts]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter ID]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter name]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster name]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster version]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster major version]
> [ INFO ] skipping: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster minor version]
> [ INFO ] skipping: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set storage_format]
> [ INFO ] ok: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add NFS storage domain]
> [ INFO ] skipping: [localhost]
> [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add glusterfs storage domain]
> [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[XML error]". HTTP response code is 400.
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[XML error]\". HTTP response code is 400."}
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XN3H2YZZ56K...
2 years, 8 months
