August 2023 - Users - oVirt List Archives

What is the status of the whole Ovirt Project?
by Arman Khalatyan 15 Oct '23

15 Oct '23

Hello everybody, What is the status of the ovirt project, would be continued it in the Rocky Linunx9? the last message is so sad: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HEKKBM6MZEKBEA… Any good news/progress there? As a happy ovirt user since 2016 should we move to an another system? Thank you beforehand, Arman.

9 14

Install Ovirt
by stevenmendoza2103＠gmail.com 04 Sep '23

04 Sep '23

Hello People; Could somebody help me with the installation of Ovirt , i am newbie , already i tried many times do the proccess but i cant goal it

2 1

error
by hillaryzunigacastro1123＠gmail.com 31 Aug '23

31 Aug '23

what would be the solution ? Errors during downloading metadata for repository 'centos-ovirt45': - Status code: 503 for http://mirrorlist.centos.org/?release-8-stream&arch-x86_64&repo-virt-ovirt- 45 (IP: 18.225.36.18) - Status code: 503 for http://mirrorlist.centos.org/?release-8-stream&arch=x86_64&repo-virt-ovirt- 45 (IP: 35.180.43.213) Error: Failed to download metadata for repo 'centos-ovirt45': Cannot prepare internal mirrorlist: St atus code: 503 for http://mirrorlist.centos.org/?release-8-stream&arch-x86_64&repo-virt-ovirt-… (IP 35.180.43.213)

2 1

I login web interface using IP with Internal Server Error
by javin158＠163.com 31 Aug '23

31 Aug '23

Hi, I’m using ovirt version 4.5.4, and I want to log in to the web interface using an IP address. After modifying the field SSO_ALTERNATE_ENGINE_FQDNS in the configuration file /etc/ovirt-engine/engine.conf.d/11-setup-sso.conf to an IP address, I was able to open the URL https://172.20.52.245/ovirt-engine/. However, when I tried to log in as a user, I encountered the following error: " Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error. More information about this error may be available in the server error log. " I got error message from file /var/log/httpd/ssl_error_log: " [Wed Aug 30 10:42:48.825547 2023] [auth_openidc:error] [pid 64405:tid 139916067432192] [client 172.20.50.110:60017] oidc_authenticate_user: the URL hostname (ovirt-engine-01.labl.local) of the configured OIDCRedirectURI does not match the URL hostname of the URL being accessed (172.20.52.245): the "state" and " session" cookies will not be shared between the two!, referer: https://172.20.52.245/ovirt-engine/" Can you please point me in some direction to try to fix this? Thanks, and best regards cph

1 0

Re: Separate NFS Domain Storage for each Domain Storage
by Mikhail Po 30 Aug '23

30 Aug '23

For the initial installation of the operating system in a virtual machine, you need to mount an ISO image that is located in the storage domain of the NFS type (ISO function). If the NFS server is located within a high-speed LAN (one data center), then there are no problems, if you need to connect an image to a remote site (second data center) with slow channels, then difficulties arise.

3 3

Is there any way to provide only url (noVNC) without ovirt account?
by rak kim 29 Aug '23

29 Aug '23

I would like to provide the vm's console to my customers. I want to provide only url without ovirt account, is there any way? I usally use novnc but if other ways can, ill use

1 0

Separate NFS Domain Storage for each Domain Storage
by Mikhail Po 28 Aug '23

28 Aug '23

Is it possible in oVirt 4.3 for each Data Center to create a separate ISO Domain Storage, with NFS type, on one of the hosts belonging to this Data Center? For example, in the case of connecting a remote site in a separate Data Center to an existing Engine through slow network links.

2 1

engine-config -s UserSessionTimeOutInterval=X problem
by marek 28 Aug '23

28 Aug '23

ovirt 4.5.4, standalone engine, centos 8 stream [root@ovirt ~]# engine-config -g UserSessionTimeOutInterval Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false UserSessionTimeOutInterval: 30 version: general [root@ovirt ~]# engine-config -s UserSessionTimeOutInterval=60 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Cannot set value 60 to key UserSessionTimeOutInterval. any ideas where is the problem? Marek

5 6

Node Intel I7 13700 Raptor Lake
by sarpi17＠gmail.com 22 Aug '23

22 Aug '23

Hello, how are you? I have an Ovirt Engine Version 4.4.10 installed. I installed Ovirt Node on a computer with an Intel I7 13700 Raptor Lake processor. When I want to link that node with the Engine I get the error that the CPU family is not correct, I tried with several of those that appear, but none is correct and Raptor Lake does not appear. What can I do? Thanks

1 0

oVirt 4.5.5 VDSM error
by cmicarelli＠fdmservices.it 22 Aug '23

22 Aug '23

HI guys, I've installed a new ovirt stack 4.5 and everything works fine. After the upgrade to 4.5.5 VDSM doesn't work correctly --------------------------------------------------------------------------------------- [root@lce8cdpnode2 opt]# systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2023-08-22 14:34:44 CEST; 19min ago Process: 3332 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 3413 (vdsmd) Tasks: 56 (limit: 823896) Memory: 312.3M CGroup: /system.slice/vdsmd.service ├─3413 /usr/bin/python3 /usr/libexec/vdsm/vdsmd └─4704 /usr/libexec/ioprocess --read-pipe-fd 59 --write-pipe-fd 58 --max-threads 10 --max-queued-requests 10 Aug 22 14:34:44 lce8cdpnode2..local vdsmd_init_common.sh[3332]: vdsm: Running test_space Aug 22 14:34:44 lce8cdpnode2..local vdsmd_init_common.sh[3332]: vdsm: Running test_lo Aug 22 14:34:44 lce8cdpnode2..local systemd[1]: Started Virtual Desktop Server Manager. Aug 22 14:34:47 lce8cdpnode2..local vdsm[3413]: WARN MOM not available. Error: [Errno 2] No such file or directory Aug 22 14:34:47 lce8cdpnode2..local vdsm[3413]: WARN MOM not available, KSM stats will be missing. Error: Aug 22 14:41:30 lce8cdpnode2..local vdsm[3413]: WARN unhandled write event Aug 22 14:41:40 lce8cdpnode2..local vdsm[3413]: WARN unhandled write event Aug 22 14:41:43 lce8cdpnode2..local vdsm[3413]: ERROR Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Aug 22 14:41:43 lce8cdpnode2..local vdsm[3413]: ERROR Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Aug 22 14:41:45 lce8cdpnode2..local vdsm[3413]: ERROR Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc> lines 7-72/72 (END) ------------------------------ Someone have had same problem Regards C

2 1

ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials
by cmicarelli＠fdmservices.it 22 Aug '23

22 Aug '23

Hi Guys, I'm tried to install ovirt 4.5 on centos 8 stream and everytime file with this error ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials Someone could help me? Regrards

2 2

Re: Active Active Stretched Node cluster not working as expected.
by Gianluca Cecchi 22 Aug '23

22 Aug '23

On Tue, Aug 22, 2023 at 8:52 AM Gaurang Patel <gaurang.patel(a)allotgroup.com> wrote: > Suggest any other solution if any. > > > Your subject says "not working as expected" and you refer to the RHV DR guides. My comment is that in my opinion HCI architecture is not expected to work / be supported for active active DR scenarios as described there. But you can follow general GlusterFS indications for solving split brain scenarios, like this one: https://docs.gluster.org/en/main/Troubleshooting/resolving-splitbrain/ and / or submit your question also to GlusterFS mailing lists or see if here you find useful information for supported configurations, quorum setup to prevent split brain conditions, ecc: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/h… https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/h… HIH, Gianluca

1 0

Active Active Stretched Node cluster not working as expected.
by gaurang.patel＠allotgroup.com 22 Aug '23

22 Aug '23

Hi, i have a Six node Stretched HCI cluster with 2 x 3 =6 distributed Replicated bricks, want to achieve Active - Active Disaster Recovery , if my three hosts in DC power Down or Failed my DR Site Three node should take care of running virtual machines. During Split-brain scenario Ovirt-Hosted-Engine VM goes in PAUSE mode and not restarting on DR hosts and waited for long time, due to this my all production VMs are not restarting on DR Site. please guide to achieve Active-Active Disaster Recovery with 6 node stretched cluster. i referred document hosted official site but did not get success, it would be great help of anybody can extend help or guide me for the same. https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/pd… My Setup insight. root@phnode06 /]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 172.16.1.1:/gluster_bricks/data/data 58744 0 Y 1746524 Brick 172.16.1.2:/gluster_bricks/data/data 58732 0 Y 2933385 Brick 172.16.1.3:/gluster_bricks/data/data 60442 0 Y 6158 Brick phnode04..local:/gluster_bricks /data/data1 51661 0 Y 5905 Brick phnode05..local:/gluster_bricks /data/data1 52177 0 Y 6178 Brick phnode06..local:/gluster_bricks /data/data1 59910 0 Y 6208 Self-heal Daemon on localhost N/A N/A Y 5878 Self-heal Daemon on phnode03..local N/A N/A Y 6133 Self-heal Daemon on gluster01..local N/A N/A Y 6072 Self-heal Daemon on phnode05..local N/A N/A Y 5751 Self-heal Daemon on gluster02..local N/A N/A Y 5746 Self-heal Daemon on phnode04..local N/A N/A Y 4734 Task Status of Volume data ------------------------------------------------------------------------------ Task : Rebalance ID : 1c30fbfa-2707-457e-b844-ccaf2e07aa7f Status : completed Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 172.16.1.1:/gluster_bricks/engine/eng ine 53723 0 Y 1746535 Brick 172.16.1.2:/gluster_bricks/engine/eng ine 49937 0 Y 2933395 Brick 172.16.1.3:/gluster_bricks/engine/eng ine 53370 0 Y 6171 Brick phnode04..local:/gluster_bricks /engine/engine1 52024 0 Y 5916 Brick phnode05..local:/gluster_bricks /engine/engine1 51785 0 Y 6189 Brick phnode06..local:/gluster_bricks /engine/engine1 55443 0 Y 6221 Self-heal Daemon on localhost N/A N/A Y 5878 Self-heal Daemon on gluster02..local N/A N/A Y 5746 Self-heal Daemon on gluster01..local N/A N/A Y 6072 Self-heal Daemon on phnode05..local N/A N/A Y 5751 Self-heal Daemon on phnode03..local N/A N/A Y 6133 Self-heal Daemon on phnode04..local N/A N/A Y 4734 Task Status of Volume engine ------------------------------------------------------------------------------ Task : Rebalance ID : 21304de4-b100-4860-b408-5e58103080a2 Status : completed Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 172.16.1.1:/gluster_bricks/vmstore/vm store 50244 0 Y 1746546 Brick 172.16.1.2:/gluster_bricks/vmstore/vm store 51607 0 Y 2933405 Brick 172.16.1.3:/gluster_bricks/vmstore/vm store 49835 0 Y 6182 Brick phnode04..local:/gluster_bricks /vmstore/vmstore1 54098 0 Y 5927 Brick phnode05..local:/gluster_bricks /vmstore/vmstore1 56565 0 Y 6200 Brick phnode06..local:/gluster_bricks /vmstore/vmstore1 58653 0 Y 1853456 Self-heal Daemon on localhost N/A N/A Y 5878 Self-heal Daemon on gluster02..local N/A N/A Y 5746 Self-heal Daemon on gluster01..local N/A N/A Y 6072 Self-heal Daemon on phnode05..local N/A N/A Y 5751 Self-heal Daemon on phnode03..local N/A N/A Y 6133 Self-heal Daemon on phnode04..local N/A N/A Y 4734 Task Status of Volume vmstore ------------------------------------------------------------------------------ Task : Rebalance ID : 7d1849f8-433a-43b1-93bc-d674a7b0aef3 Status : completed [root@phnode06 /]# hosted-engine --vm-status --== Host phnode01..local (id: 1) status ==-- Host ID : 1 Host timestamp : 9113939 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : phnode01..local Local maintenance : False stopped : False crc32 : 00979124 conf_on_shared_storage : True local_conf_timestamp : 9113939 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=9113939 (Sat Aug 19 14:39:41 2023) host-id=1 score=3400 vm_conf_refresh_time=9113939 (Sat Aug 19 14:39:41 2023) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False --== Host phnode03..local (id: 2) status ==-- Host ID : 2 Host timestamp : 405629 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode03..local Local maintenance : False stopped : False crc32 : 0b1c7489 conf_on_shared_storage : True local_conf_timestamp : 405630 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=405629 (Sat Aug 19 14:39:43 2023) host-id=2 score=3400 vm_conf_refresh_time=405630 (Sat Aug 19 14:39:43 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode02..local (id: 3) status ==-- Host ID : 3 Host timestamp : 9069311 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode02..local Local maintenance : False stopped : False crc32 : b84baa31 conf_on_shared_storage : True local_conf_timestamp : 9069311 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=9069311 (Sat Aug 19 14:39:41 2023) host-id=3 score=3400 vm_conf_refresh_time=9069311 (Sat Aug 19 14:39:41 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode04..local (id: 4) status ==-- Host ID : 4 Host timestamp : 1055893 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode04..local Local maintenance : False stopped : False crc32 : 58a9b84c conf_on_shared_storage : True local_conf_timestamp : 1055893 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1055893 (Sat Aug 19 14:39:38 2023) host-id=4 score=3400 vm_conf_refresh_time=1055893 (Sat Aug 19 14:39:38 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode05..local (id: 5) status ==-- Host ID : 5 Host timestamp : 404187 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode05..local Local maintenance : False stopped : False crc32 : 48cf9186 conf_on_shared_storage : True local_conf_timestamp : 404187 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=404187 (Sat Aug 19 14:39:40 2023) host-id=5 score=3400 vm_conf_refresh_time=404187 (Sat Aug 19 14:39:40 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host phnode06..local (id: 6) status ==-- Host ID : 6 Host timestamp : 1055902 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : phnode06..local Local maintenance : False stopped : False crc32 : 913ff6c2 conf_on_shared_storage : True local_conf_timestamp : 1055902 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1055902 (Sat Aug 19 14:39:48 2023) host-id=6 score=3400 vm_conf_refresh_time=1055902 (Sat Aug 19 14:39:48 2023) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False [root@phnode06 /]#

3 2

Issue - Bonding 802.3ad
by Anthony Bustillos 21 Aug '23

21 Aug '23

Hello team, I have this warning when I tried, to configure two ports with boonding 802.3ad. Warning: Bond is in link aggregation mode (Mode 4 ) but no partner mac has been reported for it. At least one slave has a different aggregator id. how I can to solved this warning? best Regards

2 2

How to use Python to manage each node in batches, because I want to send some scripts to nodes in batches
by ZiYi Liu 21 Aug '23

21 Aug '23

By default, find the management in engine-web and use ssh management to restart and stop the node node. Can I get the host's secret key or other login information to log in through Python, and then send some commands or scripts?

4 4

Update path of hosted_storage domain
by francesco＠shellrent.com 18 Aug '23

18 Aug '23

Hello everyone, we have a self hosted engine environment (oVirt 4.4.5) that use a replica 2 + arbiter glusterfs. Those servers are both glusterfs nodes and oVirt node of the hosted engine. For an upgrade we followed this guide https://access.redhat.com/documentation/it-it/red_hat_hyperconverged_infras…. The planned upgrade was adding 2 new servers (node3 and node4) that would replace the existing ones. We added to oVirt cluster and the glusterfs pool the servers and moved the engine in those new hosts, without touching the underlay glusterfs configuration. After the step n° 18 ("Reboot the existing and replacement hosts.Wait until all hosts are available before continuing.") of the guide we tried to start the engine but we got error on the hosted_storage domain due to old path (the glusterfs mounted path was "node1" and the backupvolfile was "node2"). For avoiding corruption we updated the database with the correct path and mount options accordly with the new configuration edited in the file /etc/ovirt-hosted-engine/hosted-engine.conf (as written in the guide). If we try to detach the node1 brick everything stop working causing storage error. We noticed that reference of node1 is still present in the file /rhev/data-center/mnt/glusterSD/node1.ovirt.xyz:_engine/36c75f6e-d95d-48f4-9ceb-ad2895ab2123/dom_md/metadata, on both of new hosts (node3 and node4). I'll be more than glad to attach any log file needed for understanding what's going on, and thank you whoever will take time for helping me out :) Regards, Francesco

1 0

Task Configure OVN for oVirt failed to execute
by Jorge Visentini 16 Aug '23

16 Aug '23

Hi, I am trying to reinstall this host, but I cannot because of this issue. Any tips for me? Host ksmmi1r01ovirt20.kosmo.cloud installation failed. Task Configure OVN for oVirt failed to execute. Please check logs for more details: /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20230815194759-ksmmi1r01ovirt20.kosmo.cloud-63609c39-934a-4360-a822-f2f6935cc5b6.log. "stdout" : "fatal: [ksmmi1r01ovirt20.kosmo.cloud]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"10.250.156.20\", \"ksmmi1r01ovirt20.kosmo.cloud\"], \"delta\": \"0:00:02.319187\", \"end\": \"2023-08-15 19:50:05.483458\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2023-08-15 19:50:03.164271\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'ksmmi1r01ovirt20.kosmo.cloud'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: ksmmi1r01ovirt20.kosmo.cloud\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\", \" return networks[net_name]\", \"KeyError: 'ksmmi1r01ovirt20.kosmo.cloud'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: ksmmi1r01ovirt20.kosmo.cloud\"], \"stdout\": \"\", \"stdout_lines\": []}", [root@ksmmi1r01ovirt20 ~]# vdsm-tool ovn-config 10.250.156.20 ksmmi1r01ovirt20.kosmo.cloud Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py", line 117, in get_network return networks[net_name] KeyError: 'ksmmi1r01ovirt20.kosmo.cloud' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 195, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py", line 63, in ovn_config ip_address = get_ip_addr(get_network(network_caps(), net_name)) File "/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py", line 119, in get_network raise NetworkNotFoundError(net_name) vdsm.tool.ovn_config.NetworkNotFoundError: ksmmi1r01ovirt20.kosmo.cloud [root@ksmmi1r01ovirt20 ~]# vdsm-tool list-nets ovirtmgmt (default route) -- Att, Jorge Visentini +55 55 98432-9868

1 0

oVirt 4.5.5 snapshot - Migration failed due to an Error: Fatal error during migration
by Jorge Visentini 13 Aug '23

13 Aug '23

Any tips about this error? 2023-08-10 18:24:57,544-03 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[29032e83-cfaf-4d30-bcc2-df72c5358552=VM]', sharedLocks=''}' 2023-08-10 18:24:57,578-03 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: 29032e83-cfaf-4d30-bcc2-df72c5358552 Type: VMAction group MIGRATE_VM with role type USER 2023-08-10 18:24:57,628-03 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='282b69aa-8b74-4312-8cc0-9c20e01982b7', vmId='29032e83-cfaf-4d30-bcc2-df72c5358552', srcHost='ksmmi1r01ovirt18', dstVdsId='73c38b36-36da-4ffa-b17a-492fd7b093ae', dstHost='ksmmi1r01ovirt19:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', migrateEncrypted='false', consoleAddress='null', maxBandwidth='3125', parallel='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='10.250.156.19', cpusets='null', numaNodesets='null'}), log id: 5bbc21d6 2023-08-10 18:24:57,628-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] START, MigrateBrokerVDSCommand(HostName = ksmmi1r01ovirt18, MigrateVDSCommandParameters:{hostId='282b69aa-8b74-4312-8cc0-9c20e01982b7', vmId='29032e83-cfaf-4d30-bcc2-df72c5358552', srcHost='ksmmi1r01ovirt18', dstVdsId='73c38b36-36da-4ffa-b17a-492fd7b093ae', dstHost='ksmmi1r01ovirt19:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', migrateEncrypted='false', consoleAddress='null', maxBandwidth='3125', parallel='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='10.250.156.19', cpusets='null', numaNodesets='null'}), log id: 14d92c9 2023-08-10 18:24:57,631-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] FINISH, MigrateBrokerVDSCommand, return: , log id: 14d92c9 2023-08-10 18:24:57,634-03 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 5bbc21d6 2023-08-10 18:24:57,639-03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: ROUTER, Source: ksmmi1r01ovirt18, Destination: ksmmi1r01ovirt19, User: admin@ovirt @internalkeycloak-authz). 2023-08-10 18:24:57,641-03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] VM '29032e83-cfaf-4d30-bcc2-df72c5358552'(ROUTER) moved from 'MigratingFrom' --> 'Up' 2023-08-10 18:24:57,641-03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Adding VM '29032e83-cfaf-4d30-bcc2-df72c5358552'(ROUTER) to re-run list 2023-08-10 18:24:57,643-03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-13) [] Rerun VM '29032e83-cfaf-4d30-bcc2-df72c5358552'. Called from VDS 'ksmmi1r01ovirt18' 2023-08-10 18:24:57,679-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2194) [] START, MigrateStatusVDSCommand(HostName = ksmmi1r01ovirt18, MigrateStatusVDSCommandParameters:{hostId='282b69aa-8b74-4312-8cc0-9c20e01982b7', vmId='29032e83-cfaf-4d30-bcc2-df72c5358552'}), log id: 445b81e0 2023-08-10 18:24:57,681-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2194) [] FINISH, MigrateStatusVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusReturn@12cb7b1b, log id: 445b81e0 2023-08-10 18:24:57,695-03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2194) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: ROUTER, Source: ksmmi1r01ovirt18, Destination: ksmmi1r01ovirt19). 2023-08-10 18:24:57,698-03 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (EE-ManagedThreadFactory-engine-Thread-2194) [] Lock freed to object 'EngineLock:{exclusiveLocks='[29032e83-cfaf-4d30-bcc2-df72c5358552=VM]', sharedLocks=''}' ovirt-release-master-4.5.5-0.0.master.20230612064154.git0c65b0e.el9.noarch ovirt-imageio-common-2.5.1-0.202307060620.git4e5f7e0.el9.x86_64 python3-ovirt-engine-sdk4-4.6.3-0.1.master.20230324091708.el9.x86_64 ovirt-openvswitch-ovn-2.17-1.el9.noarch ovirt-openvswitch-ovn-common-2.17-1.el9.noarch ovirt-openvswitch-ovn-host-2.17-1.el9.noarch ovirt-imageio-client-2.5.1-0.202307060620.git4e5f7e0.el9.x86_64 ovirt-imageio-daemon-2.5.1-0.202307060620.git4e5f7e0.el9.x86_64 python3-ovirt-setup-lib-1.3.4-0.0.master.20220413133253.gitd32d35f.el9.noarch python3.11-ovirt-engine-sdk4-4.6.3-0.1.master.20230324091708.el9.x86_64 python3.11-ovirt-imageio-common-2.5.1-0.202307060620.git4e5f7e0.el9.x86_64 python3.11-ovirt-imageio-client-2.5.1-0.202307060620.git4e5f7e0.el9.x86_64 ovirt-ansible-collection-3.1.3-0.1.master.20230420113738.el9.noarch ovirt-vmconsole-1.0.9-2.el9.noarch ovirt-vmconsole-host-1.0.9-2.el9.noarch ovirt-openvswitch-2.17-1.el9.noarch ovirt-python-openvswitch-2.17-1.el9.noarch ovirt-openvswitch-ipsec-2.17-1.el9.noarch python3-ovirt-node-ng-nodectl-4.4.3-0.20220615.0.el9.noarch ovirt-node-ng-nodectl-4.4.3-0.20220615.0.el9.noarch ovirt-host-dependencies-4.5.0-3.1.20220510094000.git2f2d022.el9.x86_64 ovirt-hosted-engine-ha-2.5.1-0.0.master.20220707064804.20220707064802.git14b1139.el9.noarch ovirt-provider-ovn-driver-1.2.37-0.20220610132522.git62111d0.el9.noarch ovirt-hosted-engine-setup-2.7.1-0.0.master.20230414113600.git340e19b.el9.noarch ovirt-host-4.5.0-3.1.20220510094000.git2f2d022.el9.x86_64 ovirt-release-host-node-4.5.5-0.0.master.20230612064150.git0c65b0e.el9.x86_64 ovirt-node-ng-image-update-placeholder-4.5.5-0.0.master.20230612064150.git0c65b0e.el9.noarch qemu-kvm-tools-8.0.0-6.el9.x86_64 qemu-kvm-docs-8.0.0-6.el9.x86_64 qemu-kvm-common-8.0.0-6.el9.x86_64 qemu-kvm-device-display-virtio-gpu-8.0.0-6.el9.x86_64 qemu-kvm-ui-opengl-8.0.0-6.el9.x86_64 qemu-kvm-ui-egl-headless-8.0.0-6.el9.x86_64 qemu-kvm-device-display-virtio-gpu-pci-8.0.0-6.el9.x86_64 qemu-kvm-block-blkio-8.0.0-6.el9.x86_64 qemu-kvm-device-display-virtio-vga-8.0.0-6.el9.x86_64 qemu-kvm-device-usb-host-8.0.0-6.el9.x86_64 qemu-kvm-device-usb-redirect-8.0.0-6.el9.x86_64 qemu-kvm-audio-pa-8.0.0-6.el9.x86_64 qemu-kvm-block-rbd-8.0.0-6.el9.x86_64 qemu-kvm-core-8.0.0-6.el9.x86_64 qemu-kvm-8.0.0-6.el9.x86_64 libvirt-daemon-kvm-9.3.0-2.el9.x86_64 Cheers! -- Att, Jorge Visentini +55 55 98432-9868

3 4

CPU support Xeon E5345
by Mikhail Po 11 Aug '23

11 Aug '23

Is it possible to install oVirt 4.3/4.4 on ProLiant BL460c G1 with Intel Xeon E5345 processor(a)2.33GHz ? In case of a setup failure, [ERROR] is fatal:[localhost]: FAILED!=>{"Changed":false,"message":"The host was inoperable, deployment errors: code 156:Host host1.test.com disabled because the host CPU type in this case is not supported by the cluster compatibility version or is not supported at all, code 9000: Failed to check the power management configuration for the host host1.test.com ., correct accordingly and re-deploy."

2 2

Trouble restoring + upgrading to ovirt 4.5 system after host crashed
by David Johnson 09 Aug '23

09 Aug '23

Good afternoon all, We had a confluence of events hit all at once and need help desperately. Our Ovirt engine system recently crashed and is unrecoverable. Due to a power maintenance event at the data center, 1/3 of our VM's are offline. I have recent backups from the engine created with engine-backup. I installed a clean Centos 9 and followed the directions to install the ovirt-engine . After I restore the backup, the engine-setup fails on the keycloak configuration. *From clean system:* *Install: **(Observe failed scriptlet during install, but rom install still succeeds)* [root@ovirt2 administrator]# dnf install -y ovirt-engine Last metadata expiration check: 2:08:15 ago on Tue 08 Aug 2023 10:11:31 AM CDT. Dependencies resolved. ============================================================================================================================================================= Package Architecture Version Repository Size ============================================================================================================================================================= Installing: ovirt-engine noarch 4.5.4-1.el9 centos-ovirt45 13 M Installing dependencies: SuperLU x86_64 5.3.0-2.el9 epel 182 k (Snip ...) * Running scriptlet: ovirt-vmconsole-1.0.9-1.el9.noarch 60/425Failed to resolve allow statement at /var/lib/selinux/targeted/tmp/modules/400/ovirt_vmconsole/cil:539Failed to resolve AST/usr/sbin/semodule: Failed!* (Snip ...) xmlrpc-common-3.1.3-1.1.el9.noarch xorg-x11-fonts-ISO8859-1-100dpi-7.5-33.el9.noarch zziplib-0.13.71-9.el9.x86_64 Complete! *Engine-restore (no visible issues):* [root@ovirt2 administrator]# engine-backup --mode=restore --log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak --provision-all-databases --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: Downloads/engine-2023-08-06.22.00.02.bak log file: restore1.log Preparing to restore: - Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak' Restoring: - Files ------------------------------------------------------------------------------ Please note: Operating system is different from the one used during backup. Current operating system: centos9 Operating system at backup: centos8 Apache httpd configuration will not be restored. You will be asked about it on the next engine-setup run. ------------------------------------------------------------------------------ Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' - user 'ovirt_engine_history', database 'ovirt_engine_history' - user 'ovirt_engine_history_grafana' on database 'ovirt_engine_history' Restoring: - Engine database 'engine' - Cleaning up temporary tables in engine database 'engine' - Updating DbJustRestored VdcOption in engine database - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine database - Resetting HA VM status ------------------------------------------------------------------------------ Please note: The engine database was backed up at 2023-08-06 22:00:19.000000000 -0500 . Objects that were added, removed or changed after this date, such as virtual machines, disks, etc., are missing in the engine, and will probably require recovery or recreation. ------------------------------------------------------------------------------ - DWH database 'ovirt_engine_history' - Grafana database '/var/lib/grafana/grafana.db' You should now run engine-setup. Done. [root@ovirt2 administrator]# *Engine-setup :* [root@ovirt2 administrator]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log Version: otopi-1.10.3 (otopi-1.10.3-1.el9) [ INFO ] The engine DB has been restored from a backup *[ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to Keycloak database 'ovirt_engine_keycloak' using existing credentials: ovirt_engine_keycloak@localhost:5432*[ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230808124504-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination *[ ERROR ] Execution of setup failed[root@ovirt2 administrator]#* *Engine-cleanup results:* (snip) [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20230808120445-mj4eef.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230808120508-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of cleanup completed successfully [root@cen-90-tmpl administrator]# *Engine backup (restore) results:* [root@ovirt2 administrator]# engine-backup --mode=restore --log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak --provision-all-databases --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: Downloads/engine-2023-08-06.22.00.02.bak log file: restore1.log Preparing to restore: - Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak' Restoring: - Files ------------------------------------------------------------------------------ Please note: Operating system is different from the one used during backup. Current operating system: centos9 Operating system at backup: centos8 Apache httpd configuration will not be restored. You will be asked about it on the next engine-setup run. ------------------------------------------------------------------------------ Provisioning PostgreSQL users/databases: - user 'engine', database 'engine' *FATAL: Existing database 'engine' or user 'engine' found and temporary ones created - Please clean up everything and try again* Any advice would be appreciated. *David Johnson*

2 2

Trouble restoring + upgrading to ovirt 4.5 system after host crashed
by David Johnson 09 Aug '23

09 Aug '23

Update: I have confirmed the original ovirt version has an ovirt_engine_keycloak database, but the database was not backed up by the engine-backup command *David Johnson* *Director of Development, Maxis Technology* 844.696.2947 ext 702 (o) | 479.531.3590 (c) <https://www.linkedin.com/in/pojoguy/> <https://maxistechnology.com/wp-content/uploads/vcards/vcard-David_Johnson.v…> <https://maxistechnology.com/> *Follow us:* <https://www.linkedin.com/company/maxis-tech-inc/>

1 1

ovirt node NonResponsive
by carlos.mendes＠mgo.cv 09 Aug '23

09 Aug '23

Hello, I have ovirt with two nodes and one that are NonResponsive and and cant manage them because they are in Unknown state. It seems that nodes lost connection for a while with their gateway. The node (ovirt2) however is having consistent problems. The follow sequence of events is reproducible and is causing the host to enter a "NonOperational" state on the cluster: What is the proper way of restoring management? I have a two-node cluster with the ovirt manager running standlone on the virtual maachine CentOS-Stream-9 and the ovirt node running the most recent oVirt Node 4.5.4 software. I can then re-activate ovirt2, which appears as green for approximately 5 minutes and then repeats all of the above issues. What can I do to troubleshoot this?

1 0

How restore nodes ovirt UP from NonResponsive and VMs executing
by José Pascual 08 Aug '23

08 Aug '23

Hello, I have ovirt with two nodes that are NonResponsive and all the VMs are properly executing but i cant manage them because are in Unknown state. It seems that nodes lost connection for a while with their gateway. I have thought of first restarting the node where the engine is not running and trying to put in UP. Then restart the engine from within de VM to see if it starts up on this node. What is the proper way of restoring management? I Thanks, Best Regards -- Saludos, José Pascual Gallud Martínez Nombre | Dpto. Ingeniería <http://telfy.com/>

3 2

Re: Some problems with ovirt
by אריה קלטר 07 Aug '23

07 Aug '23

Hi, Sorry for the late reply I tried now to get logs for the vm, for the scenario of stucked in the middle of the power up because of no disk. The vm id is 486cea97-ed56-47d4-930b-5f85c51ad3cf, the vm name is kc26-1 [image: image.png] About the second problem, with the migration, i also attached the logs here, both from the source and from the destination server. Any clue how to solve the problems? It is רeally annoying that a lot of times when I am trying to power on the vm it hangs like that, and when the system is doing live migration for the engine vm it is always not working and I need to power it off and on manually from the cli. For both scenarios i uploaded bot the /var/log/vdsm/vdsm.log and /var/log/libvirt/qemu/<vmname>.log

1 0

Certificates expired...
by Jason P. Thomas 07 Aug '23

07 Aug '23

We're moving to a new facility and pretty much building the infrastructure out from scratch. As such, the oVirt 4.4 cluster at our old location has floated under notice because it has just worked for years. In July it seems some of the certs expired (specifically the engine apache cert) and we just noticed it. I followed a post for changing the apache cert and that allowed us to login to the engine web interface, but nothing in the interface showed as connected. VMs are still running, I even rebooted one via ssh before realizing the certificate issues. In "Events" in the engine, it was complaining about certs being expired on the hosts. I found this post to this mailing list and followed the instructions possibly in error: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NHJNETOIMSHDXM… Now the engine won't start at all and I'm afraid I'm one power outage away from complete disaster. I need to keep the old location up and functioning for another 4-6 months, so any insights would be greatly appreciated. Sincerely, Jason P. Thomas

5 11