oVirt HCI 4.4.1 DR cannot import target volume after disaster
by wodel youchi
Hi,
I have installed two HCI platforms.
Then I set up the disaster-recovery configuration
On source platform I did :
# gluster volume set all cluster.enable-shared-storage enable
# gluster system:: execute gsec_create
# gluster volume geo-replication *source* gnodesen2-01.example.com::dest
create push-pem
# gluster volume geo-replication *source* gnodesen2-01.example.com::dest
config use_meta_volume true
On target platform I did :
# gluster volume set *dest* features.shard enable
I then realized these two tests :
*1 - Sync data using the schedule_georep.py*
a - Use the : "python3 /usr/share/glusterfs/scripts/schedule_georep.py
source gnodesen2-01.example.com dest " command to sync data between the
volumes
b - After the sync is complete, execute "gluster volume dest set
features.read-only off"
c - Execute *import* (not create) *dest* volume on the target platform
manually through the Admin-UI, Result : The domain could not be added and
I get this error on vdsm log :
"
2020-08-24 10:18:05,644+0100 ERROR (jsonrpc/5) [storage.HSM] Unexpected
error (hsm:2843)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 456, in
getVersion
version = self.getMetaParam(DMDK_VERSION)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 452, in
getMetaParam
return self._metadata[key]
File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line
114, in __getitem__
return dec(self._dict[key])
File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line
225, in __getitem__
return self._metadata[key]
KeyError: 'VERSION'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2829,
in getStorageDomainsList
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 115, in
produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/glusterSD.py", line
62, in findDomain
return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 385,
in __init__
manifest = self.manifestClass(domainPath)
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 162,
in __init__
sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 352, in
__init__
version = self.getVersion()
* File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 458, in
getVersion raise
se.MetaDataKeyNotFoundError("key={}".format(DMDK_VERSION))vdsm.storage.exception.MetaDataKeyNotFoundError:
Meta Data key not found error: ('key=VERSION',)*
2020-08-24 10:18:05,644+0100 INFO (jsonrpc/5) [vdsm.api] FINISH
getStorageDomainsList return={'domlist': []}
from=::ffff:10.80.101.27,48282,
flow_id=5209d576-ee09-4de3-bc46-0c186c6ad52a,
task_id=bc93aaae-5449-4d0d-9884-f47ea542a963 (api:54)
2020-08-24 10:18:05,645+0100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC
call Host.getStorageDomains succeeded in 3.01 seconds (__init__:312)
2020-08-24 10:18:05,754+0100 INFO (jsonrpc/4) [vdsm.api] START
disconnectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password':
'********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options':
'backup-volfile-servers=10.90.101.23:10.90.101.25', 'iqn': '',
'connection': '10.90.101.21:/dest', 'ipv6_enabled': 'false', 'id':
'cc24df41-19b0-41ed-ae85-e091e40cc612', 'user': '', 'tpgt': '1'}],
options=None) from=::ffff:10.80.101.27,48282,
flow_id=5fcaf7d7-d3b6-4430-b5f9-28e6bd95656a,
task_id=73654a43-c86b-4250-95d7-3e3399e5bc72 (api:48)
2020-08-24 10:18:05,754+0100 INFO (jsonrpc/4) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/10.90.101.21:_dest (mount:215)
"
*Note :* I had to modify the line 105 from
/usr/share/glusterfs/scripts/schedule_georep.py to be able to use it with
python3 :
- ey = "_".join([func.func_name] + list(args))
+ key = "_".join([func.__name__] + list(args))
*2 - Sync data using the adminUI :*
a - Create a schedule on the adminUI and wait until the sync is complete.
b - Switch off read-only on target volume : gluster volume set dest
features.read-only off (on the target volume)
c - Execute ansible failover scripts : ansible-playbook
dr-rhv-failover.yml --tags="fail_over" command (on the ansible machine)
Result :
020-08-24 15:51:58,874+0100 INFO (jsonrpc/0) [IOProcessClient]
(3a5d378b-8043-4072-9c58-e838343d6028) Starting client (init:308)
2020-08-24 15:51:58,886+0100 INFO (ioprocess/140212) [IOProcess]
(3a5d378b-8043-4072-9c58-e838343d6028) Starting ioprocess (init:434)
2020-08-24 15:51:58,888+0100 INFO (jsonrpc/0)** [vdsm.api] FINISH
createStorageDomain error=Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',) from=::ffff:10.80.101.27,45118,
flow_id=765c5009, task_id=ed3a34f9-5e81-40f4-a592-78e585c1f937 (api:52)
2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') Unexpected error (task:880)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in
_run
return fn(args, *kargs)
File "<decorator-gen-121>", line 2, in createStorageDomain
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(args, *kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in
createStorageDomain
max_hosts=max_hosts)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 95, in
create
version)
**File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 60,
in _preCreateValidation
*raise
se.StorageDomainNotEmpty(typeSpecificArg)vdsm.storage.exception.StorageDomainNotEmpty:
Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',)2020-08-24 15:51:58,888+0100 INFO
(jsonrpc/0) [storage.TaskManager.Task]
(Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') aborting: Task is aborted:
"value=Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',) abortedcode=361" (task:1190)*
2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
createStorageDomain error=Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',) (dispatcher:83)
2020-08-24 15:51:58,889+0100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC*
call StorageDomain.create failed *(error 361) in 0.20 seconds (init:312)
2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [vdsm.api] START
disconnectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password':
'****', 'vfs_type': 'glusterfs', 'port': '', 'iqn': '', 'connection':
'gnodesen2-01.example.com:/dest', 'ipv6_enabled': 'false', 'id':
'd104177f-f391-45bc-a8a7-905da90d4e7d', 'user': '', 'tpgt': '1'}],
options=None) from=::ffff:10.80.101.27,45118,
flow_id=f8c31837-fd32-48f2-bed9-6138ea4c6bb0,
task_id=badf6391-06cd-4e11-ade5-95dd10b86af5 (api:48)
2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/gnodesen2-01.example.com:_dest (mount:215)
Any ideas?
Regards.
3 years, 8 months
oVirt node 4.4.1 deploy FQDN not reachable
by Roberto Nunin
Hi all
I have an issue while trying to deploy hyperconverged solution on three
ovirt node boxes.
ISO used is ovirt-node-ng-installer-4.4.1-2020072310.el8.iso.
When from cockpit I choose gluster deployment, I have a form where I can
insert both gluster fqdn names and public fqdn names (that is what I need,
due to distinct network cards & networks)
If I insert right names, that are resolved by nodes, I receive, anyway,
FQDN is not reachable below Host1 entries.
As already stated, these names are certainly resolved by DNS used.
Any hints about ?
--
Roberto Nunin
3 years, 8 months
Actual oVirt install questions
by David White
:)
Ok, after all of the questions I've pestered you guys with, I'm actually getting started. Thank you all for your help.
A few of questions right off the bat.
For this install, I'm installing the latest (4.4.2) onto CentOS 8.
1) What's the difference between the `ovirt-hosted-engine-setup` command and the `engine-setup` command? I ran both of them, and it seems like they do the exact same thing.
2) For either of the commands above, I'm able to get as far as the step where it asks about the storage: Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]
It's clear to me that using this install method, the storage should be setup first / independently of the installer.
So ok, I'll try the Cockpit install, since the documentation says that's the preferred method anyway. (I'm just a sucker for the command line, doing it myself, and understanding everything that I'm doing. I like to avoid GUIs if I can help it. LOL).
So I logged into the Cockpit Dashboard at {ip-address}:9090, clicked on 'Virtualization' then clicked on Hosted Engine. Here, I clicked on Hyperconvered, and since I'm just installing to a single host right now, I clicked on Run Gluster Wizard for Single Node.
So here's my 3rd question.
3) I can't get past the first step of this wizard. It is asking for the FQDN of the host (Host1).
- The documentation indicates that this is NOT the base OS, but I've tried that
- I've also tried the IP address that I intend to use for the manager (i.e. what I envision is akin to the vCenter IP address for a vSphere environment -- correct?)
In both cases, the wizard won't let me proceed, and keeps saying "Host is not added to known_hosts"
What am I missing, here? I'm familiar with the known_hosts file... but that's the VM / manager that I'm trying to create, I thought? Why does the wizard continue to give me that error message?
Sent with ProtonMail Secure Email.
3 years, 8 months
Guest agent ubuntu 20.04
by carl langlois
Hi,
This may not be the right place to ask but any of you is using Ubuntu 20.04
guest. I have noticed that the guest agent is not present in the repo..
Regards
Carl
3 years, 8 months
Is oVirt suited for supporting long-lived VMs than OpenStack?
by yam yam
Hi guys,
I'm wondering why people say oVirt is better suited for traditional workloads like long-lived VMs than OpenStack.
does oVirt have some additional features for it?
Of course, VMs in OpenStack might be intentionally killed according to resource pressure if auto-scaling is used.
but otherwise, I guess OpenStack has enough to run long-lived VMs.
thanks.
3 years, 8 months
Incremental VM backups
by Kevin Doyle
Hi
I am looking at ways to backup VM's, ideally that support incremental backups. I have found a couple of python scripts that snapshot a VM and back it up but not incremental. The question is what do you use to backup the VM's ? (both linux and windows)
Thanks
Kevin
3 years, 8 months
[ANN] oVirt 4.4.2 Fourth Release Candidate is now available for testing
by Lev Veyde
oVirt 4.4.2 Fourth Release Candidate is now available for testing
The oVirt Project is pleased to announce the availability of oVirt 4.4.2
Fourth Release Candidate for testing, as of August 20th, 2020.
This update is the second in a series of stabilization updates to the 4.4
series.
Important notes before you try it
Please note this is a pre-release build.
The oVirt Project makes no guarantees as to its suitability or usefulness.
This pre-release must not be used in production.
Installation instructions
For installation instructions and additional information please refer to:
https://ovirt.org/documentation/
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 8.2 or newer
* CentOS Linux (or similar) 8.2 or newer
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 8.2 or newer
* CentOS Linux (or similar) 8.2 or newer
* oVirt Node 4.4 based on CentOS Linux 8.2 (available for x86_64 only)
See the release notes [1] for installation instructions and a list of new
features and bugs fixed.
Notes:
- oVirt Appliance is already available for CentOS Linux 8
- oVirt Node NG is already available for CentOS Linux 8
Additional Resources:
* Read more about the oVirt 4.4.2 release highlights:
http://www.ovirt.org/release/4.4.2/
* Get more oVirt project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.4.2/
[2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/
--
Lev Veyde
Senior Software Engineer, RHCE | RHCVA | MCITP
Red Hat Israel
<https://www.redhat.com>
lev(a)redhat.com | lveyde(a)redhat.com
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
3 years, 8 months
VMware import : older SSL version problems
by Philip Brown
using ovirt 4.4.1 to attempt to import from VMware, I have a problem with older SSL ciphers:
libvirt.libvirtError: internal error: curl_easy_perform() returned an error: SSL connect error (35) : error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
where can I tell it "shut up and take the data"?
I found on the web a generic openssl reference that supposedly I should be able to edit
/etc/pki/tls/openssl.cnf
and add
MinProtocol=TLSv1.0
CipherString = DEFAULT@SECLEVEL=1
but that didnt help. its still complaining about the SSL protocol.
--
Philip Brown| Sr. Linux System Administrator | Medata, Inc.
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbrown(a)medata.com| www.medata.com
3 years, 8 months
ovirt-imageio : can't upload / download
by Michael Jones
Hi,
I have image upload/download working on some older ovirt servers where
it still has the split daemon/proxy...
on one newer one this feature is not working;
software in use:
CentOS 8
ovirt-engine-4.4.1.10-1.el8.noarch
ovirt-imageio-client-2.0.9-1.el8.x86_64
ovirt-imageio-daemon-2.0.9-1.el8.x86_64
ovirt-imageio-common-2.0.9-1.el8.x86_64
the ui test button allowed me to work out 50-vdsm.conf was setting the
wrong remote port... (was 54322, changed to 54323)
updated remote with;
[remote]
host = 0.0.0.0
port = 54323
the test now passes, but on upload or download it still fails.
Next i changed the control to be unix socket instead of tcp port 54324
(vdsm was giving an error: Image daemon is unsupported);
I looked up the error line in the vdsm code, and found it was looking
for unix socket: DAEMON_SOCK=/run/ovirt-imageio/sock
switching to sock seemed to resolve all errors in the vdsm log;
---
content of the imageio log;
no errors as far as i can see:
2020-08-18 12:49:56,109 INFO (MainThread) [server] Starting
(pid=2696562, version=2.0.9)
2020-08-18 12:49:56,109 DEBUG (MainThread) [services] Creating
remote.service on port 54323
2020-08-18 12:49:56,111 DEBUG (MainThread) [http] Prefer IPv4: False
2020-08-18 12:49:56,111 DEBUG (MainThread) [http] Available network
interfaces: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>,
6, '', ('0.0.0.0', 54323))]
2020-08-18 12:49:56,111 DEBUG (MainThread) [http] Creating server
socket with family=AddressFamily.AF_INET and type=SocketKind.SOCK_STREAM
2020-08-18 12:49:56,111 DEBUG (MainThread) [services] Securing server
(cafile=/etc/pki/vdsm/certs/cacert.pem,
certfile=/etc/pki/vdsm/certs/vdsmcert.pem,
keyfile=/etc/pki/vdsm/keys/vdsmkey.pem)
2020-08-18 12:49:56,113 INFO (MainThread) [services] remote.service
listening on ('0.0.0.0', 54323)
2020-08-18 12:49:56,113 DEBUG (MainThread) [services] Creating
local.service on socket '\x00/org/ovirt/imageio'
2020-08-18 12:49:56,113 INFO (MainThread) [services] local.service
listening on '\x00/org/ovirt/imageio'
2020-08-18 12:49:56,113 DEBUG (MainThread) [services] Creating
control.service on socket '/run/ovirt-imageio/sock'
2020-08-18 12:49:56,113 DEBUG (MainThread) [uhttp] Removing socket
'/run/ovirt-imageio/sock'
2020-08-18 12:49:56,113 INFO (MainThread) [services] control.service
listening on '/run/ovirt-imageio/sock'
2020-08-18 12:49:56,115 DEBUG (MainThread) [server] Changing ownership
of /run/ovirt-imageio to 988:984
2020-08-18 12:49:56,115 DEBUG (MainThread) [server] Changing ownership
of /var/log/ovirt-imageio/daemon.log to 988:984
2020-08-18 12:49:56,115 DEBUG (MainThread) [server] Dropping root
privileges, running as 988:984
2020-08-18 12:49:56,116 DEBUG (MainThread) [services] Starting
remote.service
2020-08-18 12:49:56,116 DEBUG (remote.service) [services]
remote.service started
2020-08-18 12:49:56,116 DEBUG (MainThread) [services] Starting
local.service
2020-08-18 12:49:56,117 DEBUG (local.service) [services] local.service
started
2020-08-18 12:49:56,117 DEBUG (MainThread) [services] Starting
control.service
2020-08-18 12:49:56,117 DEBUG (control.service) [services]
control.service started
2020-08-18 12:49:56,117 INFO (MainThread) [server] Ready for requests
2020-08-18 12:51:34,602 INFO (Thread-1) [http] OPEN client=local
2020-08-18 12:51:34,603 INFO (Thread-1) [tickets] [local] ADD
ticket={'dirty': False, 'ops': ['read'], 'filename':
'debian-10.3.0-amd64-netinst.iso.raw', 'size': 351272960, 'sparse':
True, 'transfer_id': 'ec7b329e-3ef1-448d-ac73-1e5e1b4c0ef9', 'uuid':
'17db8f9f-3280-4e7c-8fbf-e5326e1f98be', 'timeout': 300, 'url':
'file:///rhev/data-center/mnt/_var_local_ovirt_local__data/7ee98037-126b-4c79-b921-a9ff4272fa91/images/315af697-a064-4c2c-bdf6-8144a8218526/8e709a79-4da0-4d91-803e-d6ec12ec84c3'}
2020-08-18 12:51:34,603 DEBUG (Thread-1) [http] Empty request line,
client disconnected
2020-08-18 12:51:34,603 INFO (Thread-1) [http] CLOSE client=local
[connection 1 ops, 0.000842 s] [dispatch 1 ops, 0.000305 s]
---
vdsm log for transfer id: ec7b329e-3ef1-448d-ac73-1e5e1b4c0ef9
no errors as far as i can see...
2020-08-18 12:51:34,602+0000 INFO (jsonrpc/5) [vdsm.api] START
add_image_ticket(ticket={'dirty': False, 'ops': ['read'], 'filename':
'debian-10.3.0-amd64-netinst.iso.raw', 'size': 351272960, 'sparse':
True, 'transfer_id': 'ec7b329e-3ef1-448d-ac73-1e5e1b4c0ef9', 'uuid':
'17db8f9f-3280-4e7c-8fbf-e5326e1f98be', 'timeout': 300, 'url':
'file:///rhev/data-center/mnt/_var_local_ovirt_local__data/7ee98037-126b-4c79-b921-a9ff4272fa91/images/315af697-a064-4c2c-bdf6-8144a8218526/8e709a79-4da0-4d91-803e-d6ec12ec84c3'})
from=::ffff:37.187.146.167,58882,
flow_id=19204d8c-ffe6-4c45-81fe-96e07097d519,
task_id=441bf5ea-1e93-4b5b-aeae-f0e52b43cf62 (api:48)
2020-08-18 12:51:34,603+0000 INFO (jsonrpc/5) [vdsm.api] FINISH
add_image_ticket return=None from=::ffff:37.187.146.167,58882,
flow_id=19204d8c-ffe6-4c45-81fe-96e07097d519,
task_id=441bf5ea-1e93-4b5b-aeae-f0e52b43cf62 (api:54)
2020-08-18 12:51:34,603+0000 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer]
RPC call Host.add_image_ticket succeeded in 0.00 seconds (__init__:312)
---
engine log:
urn [status=Status [code=0, message=Done]], log id: 7ac2f68b
2020-08-18 12:51:34,604Z INFO
[org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand]
(default task-6) [19204d8c-ffe6-4c45-81fe-96e07097d519] Started transfer
session with ticket id 17db8f9f-328
0-4e7c-8fbf-e5326e1f98be, timeout 300 seconds
2020-08-18 12:51:34,604Z INFO
[org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand]
(default task-6) [19204d8c-ffe6-4c45-81fe-96e07097d519] Adding image
ticket to ovirt-imageio-proxy, id 17db8
f9f-3280-4e7c-8fbf-e5326e1f98be
2020-08-18 12:51:34,604Z ERROR
[org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand]
(default task-6) [19204d8c-ffe6-4c45-81fe-96e07097d519] Failed to add
image ticket to ovirt-imageio-proxy: C
onnection refused (Connection refused): java.lang.RuntimeException:
Connection refused (Connection refused)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.ImageioClient.getConnection(ImageioClient.java:114)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.ImageioClient.putTicket(ImageioClient.java:65)
is there a way for me to validate where engine thinks the proxy is? the
test button passed after changing remote port to 54323...
any help much appreciated.
Kind Regards,
Michael Jones
3 years, 8 months
Max running ansible-playbook
by Tommaso - Shellrent
Hi to all.
on our angine we always see max 2 ansible-playbook running on the same time.
How can we ingrease this value?
Regards.
--
--
Shellrent - Il primo hosting italiano Security First
*Tommaso De Marchi*
/COO - Chief Operating Officer/
Shellrent Srl
Via dell'Edilizia, 19 - 36100 Vicenza
Tel. 0444321155 <tel:+390444321155> | Fax 04441492177
3 years, 8 months
