June 2024 - Users - oVirt List Archives

How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Derek Atkins 28 Jun '24

28 Jun '24

Hi, I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs: (remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>) I am 99.99% sure this is because the old certs use SHA1. I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate. All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM). So.... Is there a command-line way to re-enroll manually and update the host certs? Or some other way to get all the leftover certs renewed? Thanks, -derek [1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself. -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

8 16

python-nmstate 2.2.27 and ovirt network setup
by Bill James 28 Jun '24

28 Jun '24

We have a ovirt-host-4.5.0-3.el9.x86_64 cluster. I just recently ran yum update and it upgraded python-nmstate from python3-libnmstate-2.0.0-2.el9.noarch to python3-libnmstate- 2.2.27-2.el9_4.x86_64. This broken ovirt's ability to update the network config: 2024-06-27 13:22:39,716-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFa ctory-engine-Thread-24) [ea3bc347-156d-4631-a196-c93a9f79c6c1] Command 'HostSetupNetworksVDSCommand(HostName = ovirt-b2n.j2n oc.com, HostSetupNetworksVdsCommandParameters:{hostId='12cd7cbc-0f02-4474-aaa5-b29cde8b56a4', vds='Host[ovirt-b2n.j2noc.com, 12cd7cbc-0f02-4474-aaa5-b29cde8b56a4]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', network s='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='eno12429', vl an='110', vmNetwork='false', stp='false', properties='[]', ipv4BootProtocol='DHCP', ipv4Address='null', ipv4Netmask='null', ipv4Gateway='null', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}] ', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execu tion failed: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reas on': "[Errno 2] No such file or directory: ' */usr/lib/python3.9/site-packages/libnmstate/VERSION*'"}, code = -32603 that VERSION file, along with many other files, are not in 2.2.27 version of python-nmstate rpm. My fix was to downgrade the rpm. dnf install python3-libnmstate-2.0.0-2.el9.noarch I'm sure that's not the best solution as I can no longer run dnf update. Ideas?

1 0

Re: 4.5.4 Hosted-Engine: change hosted-engine storage
by Novak, Sam 27 Jun '24

27 Jun '24

I'm in the midst of deploying a new EL9 hosted-engine for an existing cluster and it has been an absolute nightmare, and I haven't found any documentation or guidance to account for this type of scenario, which is odd, considering that the EOL for Cent 8 was announced a while ago. I know this is all best effort type work, and I'm very appreciative of that effort, but it's really making me re-think my use case unfortunately. * Sam

1 0

Re: Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12'
by Novak, Sam 27 Jun '24

27 Jun '24

Yeah I has to convert all of my P12's to use AES cd /etc/pki/ovirt-engine/keys/ for E in $(ls *.p12); do openssl pkcs12 -in $E -out $E.key -nocerts -nodes -passin file:pass.txt openssl pkcs12 -in $E -out $E.crt -nokeys -nodes -passin file:pass.txt openssl pkcs12 -in ./$E.crt -inkey ./$E.key -export -out $E.test.p12 -keypbe aes-256-cbc -certpbe aes-256-cbc -passin file:pass.txt done You'll also need to make sure you set the file owners and permissions like the original files.

1 0

Re: [ovirt-announce] Re: Reminder: CentOS Stream 8 is now EOL
by Sandro Bonazzola 27 Jun '24

27 Jun '24

Sadly I have no procedure ready for it and no time for working on it. My best advice would be to migrate from a hosted engine on el8 to a stand alone engine on el9 by installing a fresh centos stream 9 engine and re-importing the backup there. Il giorno gio 27 giu 2024 alle ore 12:01 <trockey(a)uw.edu> ha scritto: > Hi Sanbro, > I have 3 production oVirt clusters running and I missed upgrading HE > before May 31, 2024. If I am reading everything that you stated correctly I > will need to point HE CentOS Stream release 8 to vault.centos.org and > then upgrade to CentOS Stream 9. > Do you have the procedures for taking a HE CentOS Stream release 8 pointed > at vault.contos.org and upgrading it to CentOS Stream 9? > _______________________________________________ > Announce mailing list -- announce(a)ovirt.org > To unsubscribe send an email to announce-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/announce@ovirt.org/message/M5HSQFLZLF… > -- Sandro Bonazzola

1 0

Problem importing storage domain from oVirt 4.3.10 to 4.5.5
by Sumit Basu 26 Jun '24

26 Jun '24

Hi, I want to import a FC storage domain from an oVirt 4.3.10 setup to a fresh installed oVirt 4.5.5. I could move domain to maintenance, detach, remove from 4.3.10 and import with activate option to 4.5.5 for one FC LUN fine. When i tried for the second FC lun - during import,activate i get "Error while executing action Attach Storage Domain: Internal Engine Error" and in engine.log i get " Query 'GetUnregisteredDisksQuery' failed: null', and on the standalone engine, i get :- 2024-06-26 07:15:07,277+05 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-102) [] An error occurred while fetching unregistered disks from Storage Domain id 'a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429' 2024-06-26 07:15:07,277+05 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-102) [] Command 'org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand' failed: Failed to retrieve unregistered disks 2024-06-26 07:15:07,277+05 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-102) [] Exception: java.lang.RuntimeException: Failed to retrieve unregistered disks I have checked using "Scan Disk" on the FC lun before exporting from 4.3.10 and there were no floating disks. I have checked if any of the VM's on the domain have any of their disks on any other storage domain - there are none. On the host used for importing the domain, i get :- 2024-06-26 07:15:04,829+0530 INFO (jsonrpc/5) [storage.hba] Scanning FC devices (hba:42) 2024-06-26 07:15:04,938+0530 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=9f03d412-56a7-4f3f-85b6-c3fd72a68817 (api:31) 2024-06-26 07:15:04,938+0530 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'7c9e748c-442b-48d0-9b35-a769d349c73b': {'code': 0, 'lastCheck': '3.7', 'delay': '0.000974057', 'valid': True, 'version': 0, 'acquired': True, 'actual': True}, 'a589fe14-f40a-4d67-afcd-ec74b07009cf': {'code': 0, 'lastCheck': '3.7', 'delay': '0.000501601', 'valid': True, 'version': 0, 'acquired': True, 'actual': True}, '6b625ecb-1cd1-43c5-b533-32000cd050ec': {'code': 0, 'lastCheck': '3.7', 'delay': '0.000570869', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=9f03d412-56a7-4f3f-85b6-c3fd72a68817 (api:37) 2024-06-26 07:15:04,968+0530 INFO (jsonrpc/5) [storage.hba] Scanning FC devices: 0.14 seconds (utils:373) 2024-06-26 07:15:04,968+0530 INFO (jsonrpc/5) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2024-06-26 07:15:07,003+0530 INFO (jsonrpc/5) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:120) 2024-06-26 07:15:07,003+0530 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices (multipath:223) 2024-06-26 07:15:07,014+0530 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices: 0.01 seconds (utils:373) 2024-06-26 07:15:07,014+0530 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache: 2.60 seconds (utils:373) 2024-06-26 07:15:07,263+0530 INFO (jsonrpc/5) [storage.volumemanifest] Creating image directory '/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4' (blockVolume:106) 2024-06-26 07:15:07,263+0530 ERROR (jsonrpc/5) [storage.volumemanifest] Unexpected error (blockVolume:110) Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/vdsm/storage/blockVolume.py", line 108, in validateImagePath os.mkdir(imageDir, 0o755) FileNotFoundError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4' 2024-06-26 07:15:07,263+0530 INFO (jsonrpc/5) [vdsm.api] FINISH prepareImage error=Image path does not exist or cannot be accessed/created: ('/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4',) from=::ffff:10.10.99.100,51116, flow_id=872d7930-e487-4fe4-8e0a-599074105d7b, task_id=523690f8-24b3-4f1f-801e-8391ebb087a2 (api:35) 2024-06-26 07:15:07,263+0530 ERROR (jsonrpc/5) [storage.taskmanager.task] (Task='523690f8-24b3-4f1f-801e-8391ebb087a2') Unexpected error (task:860) Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/vdsm/storage/blockVolume.py", line 108, in validateImagePath os.mkdir(imageDir, 0o755) FileNotFoundError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File "<decorator-gen-169>", line 2, in prepareImage File "/usr/lib/python3.9/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File "/usr/lib/python3.9/site-packages/vdsm/storage/hsm.py", line 2958, in prepareImage legality = dom.produceVolume(imgUUID, volUUID).getLegality() File "/usr/lib/python3.9/site-packages/vdsm/storage/sd.py", line 1188, in produceVolume return self.getVolumeClass()(self.mountpoint, self.sdUUID, imgUUID, File "/usr/lib/python3.9/site-packages/vdsm/storage/volume.py", line 881, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python3.9/site-packages/vdsm/storage/blockVolume.py", line 40, in __init__ volume.VolumeManifest.__init__(self, repoPath, sdUUID, imgUUID, File "/usr/lib/python3.9/site-packages/vdsm/storage/volume.py", line 67, in __init__ self.validate() File "/usr/lib/python3.9/site-packages/vdsm/storage/blockVolume.py", line 135, in validate volume.VolumeManifest.validate(self) File "/usr/lib/python3.9/site-packages/vdsm/storage/volume.py", line 99, in validate self.validateImagePath() File "/usr/lib/python3.9/site-packages/vdsm/storage/blockVolume.py", line 111, in validateImagePath raise se.ImagePathError(imageDir) vdsm.storage.exception.ImagePathError: Image path does not exist or cannot be accessed/created: ('/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4',) 2024-06-26 07:15:07,263+0530 INFO (jsonrpc/5) [storage.taskmanager.task] (Task='523690f8-24b3-4f1f-801e-8391ebb087a2') aborting: Task is aborted: "value=Image path does not exist or cannot be accessed/created: ('/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4',) abortedcode=254" (task:1165) 2024-06-26 07:15:07,264+0530 ERROR (jsonrpc/5) [storage.dispatcher] FINISH prepareImage error=Image path does not exist or cannot be accessed/created: ('/rhev/data-center/mnt/blockSD/a5a72b44-f9bc-4e5b-ab43-ca2b1d97b429/images/f054c394-2b49-4087-9e9d-0490b23e34e4',) (dispatcher:66) 2024-06-26 07:15:07,264+0530 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Image.prepare failed (error 254) in 2.85 seconds (__init__:300) Any help will be useful, we are moving from 4.3.10 on IBM servers and storage to 4.5.5 on DELL servers and storage Regards Sumit Basu

1 0

4.5.4 with Ceph only storage
by Maurice Burrows 25 Jun '24

25 Jun '24

Hey ... A long story short ... I have an existing Red Hat Virt / Gluster hyperconverged solution that I am moving away from. I have an existing Ceph cluster that I primarily use for OpenStack and a small requirement for S3 via RGW. I'm planning to build a new oVirt 4.5.4 cluster on RHEL9 using Ceph for all storage requirements. I've read many online articles on oVirt and Ceph, and they all seem to use the Ceph iSCSI gateway, which is now in maintenance, so I'm not real keen to commit to iSCSI. So my question is, IS there any reason I cannot use CephFS for both hosted-engine and as a data storage domain? I'm currently running Ceph Pacific FWIW. Cheers

5 10

After restoring the failed host and synchronizing the data, it prompts that there are unsynchronized items
by ziyi Liu 25 Jun '24

25 Jun '24

gluster volume info Volume Name: data Type: Replicate Number of Bricks: 1 x (2 + 1) = 3 Bricks: Brick1: node-gfs1:/gluster_bricks/data/data1 Brick2: node-gfs2:/gluster_bricks/data/data1 Brick3: node-gfs3:/gluster_bricks/data/data1 (arbiter) gluster volume heal data info Number of entries: 39 /var/log/glusterfs/glustershd.log The log appears client-rpc-fops_v2.c:785:client4_0_fsync_cbk] 0-data-client-0: remote operation failed. [{errno=5}, {error=Input/output error}] How should I solve these unrepaired entries

2 2

Error importing OVA on Ovirt-4.5.4 - no value for $TERM is defined
by raulmevi2＠hotmail.com 21 Jun '24

21 Jun '24

Hello, I'm facing an issue where, in a new fresh installed Oracle OLVM - Ovirt 4.5.4, self-hosted engine with NFS shared storage. That an .OVA file cannot be imported. Steps to reproduce: - Created test virtual machine in an NFS storage domain, installed it, allright working fine. - Shutdown test VM - Export test VM to a purposedly created directory /home/vdsm, whose owner is vdsm:kvm with full permissions. - Export .OVA file concludes successfully and the file is created at that location Then, in the exact same server, same datacenter. - Rename source VM so it doesn't conflict because the same name (same result if I delete the source VM) - try to import the .OVA. using the GUI. It finds successfully the file at the specified location and begins the procedure -# up to a given point, it fails without more indications at the web GUI. No more info at the "events" than "Failed to import vm Test to Datacenter". Fiddling into the logs, I was able to find in the hosted-engine VM, at /var/log/ovirt-engine/ova the log that goes by "ovirt-ova-external-data-ansible-DATE-HostFQDN-uuid.log" an error that specifiies stdout_lines" : [ "tput: No value for $TERM and no -T specified", "tput: No value for $TERM and no -T specified", "tput: No value for $TERM and no -T specified" I tinkered with the ansible role that calls a python script called "get_ova_data.py" (located at /usr/share/ovirt-engine/ansible-runner-service-project/project/roles-/ovirt-ova-external-data/) I tried to define the variable TERM into null or into xterm but then it fails in another manner complaining about that variable. By surfing the webs, it seems something calls "tput" like if it was in console mode but then it breaks. Can I have some help with this, please? Thanks

1 0

HE deployment on FC (fibre-channel) disk fails at 99% completed at final "hosted-engine --reinitialize-lockspace --force"
by Jeffrey Slapp 21 Jun '24

21 Jun '24

This identical to many others who have encountered this issue, yet nothing definitive has been suggested. The entire HE deployment nearly finishes, but after the copy of the HE VM to the shared disk, shortly after that I reach the "initialize lockspace" section and the following error occurs: 20083 2024-06-19 13:24:01,777-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'changed': True, 'stdout': '', 'stderr': 'Traceback (most recent call last):\n File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main\n return _run_ code(code, main_globals, None,\n File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code\n exec(code, run_globals)\n File "/usr/lib/p ython3.9/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File " /usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/client/client.py", line 286, in reset_lockspace\n stats = broker.get_stats_from_sto rage()\n File "/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File "/usr/lib64/python3.9/xmlrpc/client.py", line 1122, in __call__\n ret urn self.__send(self.__name, args)\n File "/usr/lib64/python3.9/xmlrpc/client.py", line 1464, in __request\n response = self.__transport.request(\n File "/usr/lib64/python3.9/ xmlrpc/client.py", line 1166, in request\n return self.single_request(host, handler, request_body, verbose)\n File "/usr/lib64/python3.9/x mlrpc/client.py", line 1178, in single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File "/usr/lib64/py thon3.9/xmlrpc/client.py", line 1291, in send_request\n self.send_content(connection, request_body)\n File "/usr/lib64/python3.9/xmlrpc/cl ient.py", line 1321, in send_content\n connection.endheaders(request_body)\n File "/usr/lib64/python3.9/http/client.py", line 1280, in end headers\n self._send_output(message_body, encode_chunked=encode_chunked)\n File "/usr/lib64/python3.9/http/client.py", line 1040, in _send _output\n self.send(msg)\n File "/usr/lib64/python3.9/http/cl ient.py", line 980, in send\n self.connect()\n File "/usr/lib/python3.9/s ite-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 76, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundErro r: [Errno 2] No such file or directory', 'rc': 1, 'cmd': ['hosted-engine', '--reinitialize-lockspace', '--force'], 'start': '2024-06-19 13:24: 01.438386', 'end': '2024-06-19 13:24:01.618227', 'delta': '0:00:00.179841', 'msg': 'non-zero return code', 'invocation': {'module_args': {'_ra w_params': 'hosted-engine --reinitialize-lockspace --force', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv' : None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Traceback (most recent call last):', ' File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main', ' return _run_code(code, main_global s, None,', ' File "/usr/l ib64/python3.9/runpy.py", line 87, in _run_code', ' exec(code, run_globals)', ' File "/usr/lib/python3.9/site-pa ckages/ovirt_hosted_engine_setup/reinitialize_lockspace.py", line 30, in <module>', ' ha_cli.reset_lockspace(force)', ' File "/usr/lib/pyt hon3.9/site-packages/ovirt_hosted_engine_ha/client/client.py", line 286, in reset_lockspace', ' stats = broker.get_stats_from_storage()', ' File "/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage', ' result = self._ proxy.get_stats()', ' File "/usr/lib64/python3.9/xmlrpc/client.py", line 1122, in __call__', ' return self.__send(self.__name, args)', ' File "/usr/lib64/python3.9/xmlrpc/client.py", line 1464, in __request', ' response = self.__transport.request(', ' File "/usr/lib64/python 3.9/xmlrpc/client.py", line 1166, in request', ' return self.single_request(host, handler, request_body, verbose)', ' File "/usr/li b64/pyt hon3.9/xmlrpc/client.py", line 1178, in single_request', ' http_conn = self.send_request(host, handler, request_body, verbose)', ' File "/ usr/lib64/python3.9/xmlrpc/client.py", line 1291, in send_request', ' self.send_content(connection, request_body)', ' File "/usr/lib64/pyt hon3.9/xmlrpc/client.py", line 1321, in send_content', ' connection.endheaders(request_body)', ' File "/usr/lib64/python3.9/http/client.py ", line 1280, in endheaders', ' self._send_output(message_body, encode_chunked=encode_chunked)', ' File "/usr/lib64/python3.9/http/client. py", line 1040, in _send_output', ' self.send(msg)', ' File "/usr/lib64/python3.9/http/client.py", line 980, in send', ' self.connect() ', ' File "/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 76, in connect', ' self.sock.connect(base64.b16de code(self.host))', 'FileNotFoundError: [Errno 2] No such file or directory'], '_ansible_no_log': False, 'attempts': 5} No errors in /var/log/messages or /var/log/sanlock.log. I have this working with iSCSI on another storage system, but can't seem to get this to work on FC. I have read that the sector sizes could possible cause this. On my iSCSI system I have [PHY-SEC:LOG-SEC] as [512:512] but on my FC system I have [4096:512]. Hoping that someone can confirm whether this is the issue or not. Interestingly all previous "initialize lockspace" phases of the install are fine, it just appears to be this final one.

2 1

New host cannot connect to master domain
by ovirt＠kirschke.de 21 Jun '24

21 Jun '24

Hi, i just installed a new host, which cannot connect to the master domain. The domain is an iscsi device on a nas, and ovirt is actually fine with it, other hosts have no problem. I'sure , I#m just overlooking something. What I see in the log is: 024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.storagepoolmemorybackend] new storage pool master version 302 and domains map {'6f018fbd-de93-4c56-880d-8ede2aad2674': 'Active', '2c870e06-6c70-45ec-b665-ce29408c8a8e': 'Active', 'a154 96dc-c241-4658-af9d-0dfe11783916': 'Active', '41012bfb-b802-4092-b699-7f5284d95c8e': 'Active'} (spbackends:417) 2024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.storagepool] updating pool 5836aaac-0030-0064-024d-0000000002e4 backend from type NoneType instance 0x7f10d667fb70 to type StoragePoolMemoryBackend instance 0x7f10702d7408 (sp:149) 2024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.storagepool] Connect host #2 to the storage pool 5836aaac-0030-0064-024d-0000000002e4 with master domain: a15496dc-c241-4658-af9d-0dfe11783916 (ver = 302) (sp:699) 2024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.storagedomaincache] Clearing storage domain cache (sdc:182) 2024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2024-04-16 11:10:06,320-0400 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2024-04-16 11:10:06,456-0400 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices: 0.14 seconds (utils:373) 2024-04-16 11:10:06,456-0400 INFO (jsonrpc/3) [storage.hba] Scanning FC devices (hba:42) 2024-04-16 11:10:06,481-0400 INFO (jsonrpc/3) [storage.hba] Scanning FC devices: 0.03 seconds (utils:373) 2024-04-16 11:10:06,481-0400 INFO (jsonrpc/3) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2024-04-16 11:10:08,498-0400 INFO (jsonrpc/3) [storage.multipath] Waited 2.02 seconds for multipathd (tries=2, ready=2) (multipath:122) 2024-04-16 11:10:08,498-0400 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices (multipath:223) 2024-04-16 11:10:08,499-0400 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2024-04-16 11:10:08,499-0400 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache: 2.18 seconds (utils:373) 2024-04-16 11:10:08,499-0400 INFO (jsonrpc/3) [storage.storagedomaincache] Looking up domain a15496dc-c241-4658-af9d-0dfe11783916 (sdc:154) 2024-04-16 11:10:08,536-0400 INFO (jsonrpc/3) [storage.storagedomaincache] Looking up domain a15496dc-c241-4658-af9d-0dfe11783916: 0.04 seconds (utils:373) 2024-04-16 11:10:08,537-0400 INFO (jsonrpc/3) [vdsm.api] FINISH connectStoragePool error=Cannot find master domain: 'spUUID=5836aaac-0030-0064-024d-0000000002e4, msdUUID=a15496dc-c241-4658-af9d-0dfe11783916' from=::ffff:10.2.0.4,44914, flo w_id=44d9e674, task_id=32f0ea2a-044b-4a81-a12e-a269e59a802b (api:35) 2024-04-16 11:10:08,537-0400 ERROR (jsonrpc/3) [storage.taskmanager.task] (Task='32f0ea2a-044b-4a81-a12e-a269e59a802b') Unexpected error (task:860) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 1550, in setMasterDomain domain = sdCache.produce(msdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('a15496dc-c241-4658-af9d-0dfe11783916',) During handling of the above exception, another exception occurred: .... The domain a15496dc-c241-4658-af9d-0dfe11783916 definitely exist, and works on other systems. THe new host can acces the iscsi targets, and i am able to log into them. Any one who knows, which blinder I have to remove, so I see the actual problem? Thanks and best regards Steffen

1 1

Admin interface broken
by ovirt＠kirschke.de 21 Jun '24

21 Jun '24

Hi i tried to migrate my ovirt engine form a centos 8 to a 9 machine. So, I made a backup, created and configured a new machine, stopped the old one, started the new one with identical network configuration, restored the backup and tried engine-setup to set up the new site, exping it to work. It did not, but I thought, no prob, I still have my old one. Now, going back to the old mache, when logging into Admin Portal, I get error messages: - Error while executing action: A Request to the Server failed: Type 'org.ovirt.engine.core.common.queries.QueryType' was not assignable to 'com.google.gwt.user.client.rpc.IsSerializable' and did not have a custom field serializer. For security purposes, this type will not be deserialized. - Error while executing query: null Each in a Box, which I clicked away getting the messages toggeling. After a while, the abckground of the admin Portal's dashboad is becomes visible, but remains empty, with the 'loading' spinner. In the serverlog I only see the following exveption: Caused by: com.google.gwt.user.client.rpc.SerializationException: Type 'org.ovirt.engine.core.common.queries.QueryType' was not assignable to 'com.google.gwt.user.client.rpc.IsSerializable' and did not have a custom field serializer. For security purposes, this type will not be deserialized. at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.LegacySerializationPolicy.validateDeserialize(LegacySerializationPolicy.java:128) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:676) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.core.java.util.Collection_ServerCustomFieldSerializerBase.deserialize(Collection_ServerCustomFieldSerializerBase.java:38) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserialize(ArrayList_ServerCustomFieldSerializer.java:40) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:54) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.core.java.util.ArrayList_ServerCustomFieldSerializer.deserializeInstance(ArrayList_ServerCustomFieldSerializer.java:33) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeImpl(ServerSerializationStreamReader.java:887) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserialize(ServerSerializationStreamReader.java:687) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.readObject(ServerSerializationStreamReader.java:592) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader$ValueReader$8.readValue(ServerSerializationStreamReader.java:149) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.impl.ServerSerializationStreamReader.deserializeValue(ServerSerializationStreamReader.java:434) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RPC.decodeRequest(RPC.java:312) ... 64 more Which , of course I do not understand, what it tries to tell me. The VM-Portal is still reachable, and seems to work. Has anybody had an effect like that, or, even better, an idea, what I am doing wrong and how I can fix that, or where I can look to find the main cause for the bahaviour- Forgot to mention: Desperate, as I am, I also tried to upgrade to 4.5.7-master, which worked without any problem or warning, but the effect is still the same. Thanks and best regards Steffen

1 0

Ovirt4.5 installation failure on Rocky 8
by 盛家杰 20 Jun '24

20 Jun '24

Hello Ovit Devs, I am new to ovirt and am trying to install Ovirt4.5 on Rocky8 machine, and I am stuck at the first step, enable ovirt engine repo. 1. I follow the manual to do some yum repo configuration staff, with bash code in "Rocky" part , https://www.ovirt.org/download/install_on_rhel.html # On Rocky there's an issue with centos-release-nfv package from extras dnf install -y dnf-plugins-core dnf config-manager --set-disabled extras 2. I try to install ovirt4.5 with "dnf install -y centos-release-ovirt45", but it fails with "unable to find a match". I find centos-release-ovirt45 is listed in extras repo and extras repo is disabled. If extras repo needs to be disabled, how can I install centos-release-ovirt45? Any suggestions would be highly appreciated! Jiajie

1 0

How to access the ovirt node individually?
by De Lee 19 Jun '24

19 Jun '24

Hi, I've installed the ovirt manager on top of the oVirt node. For some reason the node went to offline and unable to start the ovrit manager. Do we have option to enable the auto start of the ovrit manager and can I able to connect the console of the ovirt node try to power on the VM?

2 1

Re: ovirt keycloak intergration user search not working
by Kevin Köllmann 19 Jun '24

19 Jun '24

We also noticed this and are just adding them by knowing the full username. A functioning search would be helpful though.

1 0

Please tell me how to deal with the error
by d_hashima＠sagaidc.co.jp 19 Jun '24

19 Jun '24

I'm currently installing oVirt45 self-host on RHEL9. Installation fails due to the following error. Could you please give me some advice? error contents: "Failed to download metadata for repo 'centos-ceph-pacific': Cannot prepare internal mirrorlist: No URLs in mirrorlist"

3 3

Add new host fail and reinstall fail
by max.tseng＠extremedata.com.tw 19 Jun '24

19 Jun '24

Hi Sir, I want crate new host server , always get error message 2024-06-19 14:52:59,440+08 WARN [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[79ddc832]-nio2-thread-8) globalRequest(ClientConnectionService[ClientSessionImpl[root@/172.16.1.40:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported

1 0

Out-of-sync - Host Network's configurations differ from DC
by munnadawood＠gmail.com 18 Jun '24

18 Jun '24

Hello Team, We had a power outage last week and we started observing the error on our oVirt cluster, where all the hosts went out-of-sync. When I select my ovirtmgmt interface it shows the out-of-sync symbol and shows IPv4 gateway ( Host - Null, DC - 10.xx..xx.xx) Same with my VM traffic network, it shows the out-of-sync symbol and shows IPv4 gateway ( Host - Null, DC - 10.xx..xx.xx) Please help me with my troubleshooting as we lost access to VM's access after our power outage. Thanks, Dawood.

2 2

Deployment Failure with oVirt Hosted Engine on CentOS Stream 9
by khaltarabien＠gmail.com 17 Jun '24

17 Jun '24

Hello, I've encountered a failure while trying to deploy the oVirt Hosted Engine using the oVirt Node NG Installer on a system running CentOS Stream 9. Here are the details: Environment: OS: CentOS Stream 9 (oVirt Node 4.5.5) oVirt Version: 4.5.5 Installer ISO: oVirt Node NG Installer 4.5.5-2023113015 for el9 Issue: During the deployment of the hosted engine, the process failed with several error messages, notably failing to download metadata for the 'centos-ceph-pacific' repository due to no URLs in the mirrorlist, and a failure deploying the engine on the local VM. Error Messages: Failed to download metadata for repo 'centos-ceph-pacific': Cannot prepare internal mirrorlist: No URLs in mirrorlist There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch. Deprecation warning for the Python crypt module. Steps Taken: Attempted deployment using the hosted-engine --deploy command. Used the specified installer ISO. Seeking Help: Has anyone faced similar issues, or does anyone have suggestions on how to resolve the mirrorlist problem and the deployment issue? Any help or guidance would be greatly appreciated!ERY, status: NOERROR, id: 52241 2024-06-13 00:07:19,621+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost -> 192.168.222.192]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'centos-ceph-pacific': Cannot prepare internal mirrorlist: No URLs in mirrorlist", "rc": 1, "results": []} 2024-06-13 00:07:48,484+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} 2024-06-13 00:07:49,088+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:238 b'[DEPRECATION WARNING]: Encryption using the Python crypt module is deprecated. \n' 2024-06-13 00:07:49,088+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:238 b'The Python crypt module is deprecated and will be removed from Python 3.13. \n' 2024-06-13 00:07:49,089+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:238 b'Install the passlib library for continued encryption functionality. This \n' 2024-06-13 00:07:49,089+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:238 b'feature will be removed in version 2.17. Deprecation warnings can be disabled \n' 2024-06-13 00:07:49,090+0000 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:238 b'by setting deprecation_warnings=False in ansible.cfg.\n' 2024-06-13 00:07:49,094+0000 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Closing up': Failed executing ansible-playbook 2024-06-13 00:08:11,089+0000 ERROR otopi.plugins.gr_he_common.core.misc misc._terminate:164 Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.

3 4

problem on dnf update
by g.vasilopoulos＠uoc.gr 17 Jun '24

17 Jun '24

Hi I tried dnf update today and the result was [root@kmanage ~]# dnf update Last metadata expiration check: 3:42:36 ago on Mon 17 Jun 2024 08:47:59 AM EEST. Error: Problem 1: package python3-boto3-1.18.58-1.el9s.noarch from @System requires (python3.9dist(botocore) < 1.22 with python3.9dist(botocore) >= 1.21.58), but none of the providers can be installed - cannot install both python3-botocore-1.31.62-1.el9.noarch from appstream and python3-botocore-1.21.58-1.el9s.noarch from @System - cannot install both python3-botocore-1.21.58-1.el9s.noarch from ovirt-master-centos-stream-openstack-yoga-testing and python3-botocore-1.31.62-1.el9.noarch from appstream - cannot install the best update candidate for package python3-botocore-1.21.58-1.el9s.noarch - cannot install the best update candidate for package python3-boto3-1.18.58-1.el9s.noarch Problem 2: package python3-pyngus-2.3.0-8.el9s.noarch from @System requires python3.9dist(python-qpid-proton), but none of the providers can be installed - cannot install both python3-qpid-proton-0.39.0-2.el9s.x86_64 from ovirt-master-centos-stream-opstools-collectd5-testing and python3-qpid-proton-0.35.0-2.el9s.x86_64 from @System - cannot install both python3-qpid-proton-0.35.0-2.el9s.x86_64 from ovirt-master-centos-stream-openstack-yoga-testing and python3-qpid-proton-0.39.0-2.el9s.x86_64 from ovirt-master-centos-stream-opstools-collectd5-testing - cannot install the best update candidate for package python3-qpid-proton-0.35.0-2.el9s.x86_64 - cannot install the best update candidate for package python3-pyngus-2.3.0-8.el9s.noarch Problem 3: package python3-oslo-messaging-12.13.3-1.el9s.noarch from @System requires python3-pyngus, but none of the providers can be installed - package python3-pyngus-2.3.0-8.el9s.noarch from @System requires python3.9dist(python-qpid-proton), but none of the providers can be installed - package python3-pyngus-2.3.0-8.el9s.noarch from ovirt-master-centos-stream-openstack-yoga-testing requires python3.9dist(python-qpid-proton), but none of the providers can be installed - package python3-qpid-proton-0.35.0-2.el9s.x86_64 from @System requires qpid-proton-c(x86-64) = 0.35.0-2.el9s, but none of the providers can be installed - package python3-qpid-proton-0.35.0-2.el9s.x86_64 from ovirt-master-centos-stream-openstack-yoga-testing requires qpid-proton-c(x86-64) = 0.35.0-2.el9s, but none of the providers can be installed - cannot install both qpid-proton-c-0.39.0-2.el9s.x86_64 from ovirt-master-centos-stream-opstools-collectd5-testing and qpid-proton-c-0.35.0-2.el9s.x86_64 from @System - cannot install both qpid-proton-c-0.35.0-2.el9s.x86_64 from ovirt-master-centos-stream-openstack-yoga-testing and qpid-proton-c-0.39.0-2.el9s.x86_64 from ovirt-master-centos-stream-opstools-collectd5-testing - cannot install the best update candidate for package qpid-proton-c-0.35.0-2.el9s.x86_64 - cannot install the best update candidate for package python3-oslo-messaging-12.13.3-1.el9s.noarch (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

1 0

Incremental backup in Ovirt 4.5.7-el9 and 4.5.6 -el8 not working...
by rvks＠protonmail.com 15 Jun '24

15 Jun '24

Hi All, We are trying out ovirt (self-hosted engine) to replace Vmware and it's important to test the Incremental backup but so far we had no luck to make it work. Full backup working fine but subsequent incrementals are not happening, instead it's doing full backup. Using REST api but we can't enable incremental on disk ( DiskBackupMode enum). Any tips and help appreciated..... thanks, rvks

1 0

oVirt engine installation
by andremi001＠gmail.com 14 Jun '24

14 Jun '24

Hi, I'm trying to deploy oVirt engine self hosted, from my Centos 9 stream node, but when I try to run the deploy I always got the error: "FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'centos-ceph-pacific': Cannot prepare internal mirrorlist: No URLs in mirrorlist", "rc": 1, "results": []}" I thinks it is correlated with the end of life of Centos 8 stream. How can I fix it?

2 1

oVirt 4.5.5 Fresh Installation
by Clemilton 14 Jun '24

14 Jun '24

Hope this helps the project to get better; [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "hostname 'host01' doesn't uniquely match the interface 'eno1' selected for the management bridge; it matches also interface with IP ['fe80::5a8a:5aff:fe68:4981%2']. Please make sure that the hostname got from the interface for the management network resolves only there.\n"} - THIS IS NOT MENSIONED (AT LEAST I DIDNT SEE ANYWHERE) IN THE DOCUMENTATION. [ INFO ] TASK [ovirt.ovirt.engine_setup : Check if Engine health page is up] [ INFO ] ok: [localhost -> 192.168.222.43] - THIS IS NOT THE RIGHT IP ADDRESS. THIS NETWORK DOES NOT EXIST [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up] - 25 MINUTES WAITING... [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} - COURSE! THE SCRIPT IGNORED THE IP ADDRESS I'VE PUT [ ERROR ] b'[DEPRECATION WARNING]: Encryption using the Python crypt module is deprecated. \n' [ ERROR ] b'The Python crypt module is deprecated and will be removed from Python 3.13. \n' [ ERROR ] b'Install the passlib library for continued encryption functionality. This \n' [ ERROR ] b'feature will be removed in version 2.17. Deprecation warnings can be disabled \n' [ ERROR ] b'by setting deprecation_warnings=False in ansible.cfg.\n' [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook

2 1

Reminder: CentOS Stream 8 is now EOL
by Sandro Bonazzola 13 Jun '24

13 Jun '24

Hi, just a reminder that CentOS Stream 8 reached its end of life at the end of last month on Friday, May 31, 2024. As it has been announced by CentOS project[1], all the CentOS Stream 8 repositories are going to be archived at vault.centos.org . Who didn't update yet to CentOS Stream 9 will probably hit several errors related to this archiving: you'll need to manually adjust the repository url to point to the new location at vault.centos.org. The archiving is going to happen today [2] with the decommissioning of the nodes used by the CentOS Community Build System used for oVirt builds for Stream 8 within the CentOS Virtualization SIG. At this point, the recommendation is to upgrade to CentOS Stream 9 (or derived) as soon as practical. For those who are pushing changes to the oVirt project, please note GitHub actions may start to fail on CentOS Stream 8 (and with that, the builds for oVirt Node and oVirt Appliance). It's up to you to decide if you want to change the config there to use a different el8 distribution for running the tests or just turn off el8 and rely only on CentOS Stream 9 tests. [1] https://lists.centos.org/hyperkitty/list/announce@lists.centos.org/thread/D… [2] https://lists.centos.org/hyperkitty/list/devel@lists.centos.org/thread/P3HD… -- Sandro Bonazzola

6 9

No valid authentication mechanism found
by Mathieu Valois 13 Jun '24

13 Jun '24

Hi everyone, we've configured oVirt 4.5.5 with KeyCloak OIDC using an LDAP (FreeIPA) backend. When we wish to login on the portal, i.e https://example.org/ovirt-engine, using the "Administration Portal" or login directly with the button in the upper-right corner, it works (we get redirected to keycloack and so on). However, if we first try to login by clicking on the "VM portal", it fails. We get redirected back to the portal having a "No valid authentication mechanism found" error, as seen on the following screenshot. Then, we're stuck until we clear the cache and cookies of the browser, even when clicking on "Admin portal". Backend logs only show this : 2024-06-04 11:56:24,319-04 ERROR [org.ovirt.engine.core.sso.service.SsoService] (default task-86) [] Aucun mécanisme d'authentification valide trouvé. which translates to the message seen on the screenshot. Do you get any lead to resolve such an issue? Thanks for your kind support, Mathieu Valois Ingénieur sécurité, systèmes et réseaux Bureau Caen : Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré : Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | https://teicee.com?pk_campaign=Email https://www.facebook.com/teicee https://www.linkedin.com/company/t-c-e

1 0