- Users - oVirt List Archives

Re: what is "use host" field in storage domain creation??
by Strahil 30 Dec '19

30 Dec '19

I think SPM is not doing initial setup - just other tasks - like snapshots creation/deletion. Best Regards, Strahil NikolovOn Dec 30, 2019 11:57, yam yam <hyunooudy(a)gmail.com> wrote: > > isn't it SPM's job? > and is the "use host" required only for setup? > > thanks for quick replying. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IRQCB37EXLOHQ…

1 0

what is "use host" field in storage domain creation??
by hyunooudy＠gmail.com 30 Dec '19

30 Dec '19

hello and happy new year~ I am wondering the role of "use host" field in storage domain creation. https://www.ovirt.org/documentation/install-guide/chap-Configuring_Storage.… above link says all communication to the storage domain is through "use host". but I can't understand inefficiently passing through that "use host" even though every host can directly access to all domains. it seems like i am misunderstanding something.

3 2

what is "use host" field in storage domain creation??
by hyunooudy＠gmail.com 30 Dec '19

30 Dec '19

hello and happy new year~ I am wondering role of "use host" field in storage domain creation. https://www.ovirt.org/documentation/install-guide/chap-Configuring_Storage.… above link says all communication to the storage domain is through "use host". but I can't understand inefficiently passing through that "use host" even though every host can directly access to all domains. it seems like i am misunderstanding something.

1 0

Cannot add Foreman as a provider
by Robert Webb 29 Dec '19

29 Dec '19

So I just setup a brand new Foreman instance and have been trying to get it setup with oVirt. I was able to install the oVirt plugins under Foreman and have the oVirt engine setup as a computer resource. Where I am running into an issue is adding Foreman as an additional provider under oVirt. In the log I get: Command 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' failed: EngineException: https://foreman.saroza-webb.internal:443/api/v2/discovered_hosts?format=jso… (Failed with error PROVIDER_FAILURE and code 5050) I have done some searching, but everything I have found seems to be from years ago and deals with oVirt 3.5.x Any thoughts?

1 0

Metrics
by gregory.day＠target.com 29 Dec '19

29 Dec '19

Hello. I'm trying to deploy the metrics store and the ansible run fails at "TASK [/usr/share/ansible/roles/oVirt.metrics/roles/oVirt.origin-on-ovirt : Collect oVirt VM facts]" I've tried adding the -vvv and ANSIBLE_DEBUG=1, but that isn't giving me any additional information on why it is failing. fatal: [localhost]: FAILED! => { "attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false } When I edit the playbook and set no_log: true, the output is shown, but the resulting log shows that the "ansible_facts.ovirt_vms[0].fqdn" dictionary key does not exist. What is the best way to work around this issue? Thanks, Greg

2 1

No virtio-win vfd oVirt 4.3.7
by Vrgotic, Marko 27 Dec '19

27 Dec '19

Dear oVIrt, I am trying to install Windows 10 on oVirt, following procedure on: https://www.ovirt.org/documentation/vmm-guide/chap-Installing_Windows_Virtu… Unfortunately, there is no option as explained in the “Installing Windows on VirtIO-Optimized Hardware”. There is just no “virtio-win.vfd” option when attaching Floppy. Is this valid. I have ISO Domain, which is not hosted by Hosted Engine, but mounte additionally. Is this required in order to have the virtio-win.vfd option? If not, please let me know what am I missing. In the meanwhile, I am working on building custom Wind 10 ISO which contains Virtio/Virtio-SCSI drivers. Kindly awaiting your reply. ----- kind regards/met vrindelijke groet Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

2 2

ovirt-host 4.4.0 Task Update system failed to execute - Running QEMU processes found, cannot upgrade Vdsm
by alexandermurashkin＠msn.com 27 Dec '19

27 Dec '19

On a regular basis, there is "Task Update system failed to execute" error in the engine.log. At the same time, the ovirt-host-mgmt-ansible-check log contains "Running QEMU processes found, cannot upgrade Vdsm" error. Is there some configuration setting that controls it? Why oVirt is trying to update ovirt-host package on the host at all if there are virtual machines running? Note that 2019-12-27 11:57:21,880-06 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 3 threads waiting for tasks. 2019-12-27 12:02:30,878-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-hostUpdatesChecker-Thread-4) [] EVENT_ID: ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Check for update of host harrier. Gathering Facts. 2019-12-27 12:02:45,990-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-hostUpdatesChecker-Thread-4) [] EVENT_ID: ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Check for update of host harrier. Install ovirt-host package if it isn't installed. 2019-12-27 12:02:55,024-06 ERROR [org.ovirt.engine.core.bll.host.HostUpgradeManager] (EE-ManagedExecutorService-hostUpdatesChecker-Thread-4) [] Failed to read host packages: Task Update system failed to execute: 2019-12-27 12:02:55,025-06 ERROR [org.ovirt.engine.core.bll.hostdeploy.HostUpdatesChecker] (EE-ManagedExecutorService-hostUpdatesChecker-Thread-4) [] Failed to check if updates are available for host 'harrier' with error message 'Task Update system failed to execute: ' 2019-12-27 12:02:55,106-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-hostUpdatesChecker-Thread-4) [] EVENT_ID: HOST_AVAILABLE_UPDATES_FAILED(839), Failed to check for available updates on host harrier with message 'Task Update system failed to execute: '. 2 # rpm -q ovirt-host ovirt-host-4.4.0-0.2.alpha.el8.x86_64 # dnf update Updating Subscription Management repositories. Last metadata expiration check: 0:11:31 ago on Fri Dec 27 13:16:55 2019. Error: Running QEMU processes found, cannot upgrade Vdsm. # dnf update ovirt-host Updating Subscription Management repositories. Extra Packages for Enterprise Linux 8 - x86_64 7.3 kB/s | 5.3 kB 00:00 GlusterFS 6 packages for x86_64 2.8 kB/s | 3.0 kB 00:01 GlusterFS 6 noarch packages 18 kB/s | 3.0 kB 00:00 virtio-win builds roughly matching what will be shipped in upcoming RHEL 8.2 kB/s | 3.0 kB 00:00 Copr repo for EL8_collection owned by sbonazzo 10 kB/s | 3.6 kB 00:00 Copr repo for gluster-ansible owned by sac 13 kB/s | 3.3 kB 00:00 Copr repo for ovsdbapp owned by mdbarroso 14 kB/s | 3.3 kB 00:00 Copr repo for nmstate-stable owned by nmstate 14 kB/s | 3.3 kB 00:00 Copr repo for NetworkManager-1.20 owned by networkmanager 14 kB/s | 3.3 kB 00:00 Latest oVirt master nightly snapshot 4.0 kB/s | 3.0 kB 00:00 Latest oVirt master additional nightly snapshot 5.0 kB/s | 3.0 kB 00:00 Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs) 14 kB/s | 4.5 kB 00:00 Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 27 kB/s | 4.1 kB 00:00 Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 36 kB/s | 4.5 kB 00:00 Dependencies resolved. Nothing to do. Complete! ------------------------- ovirt-host-mgmt-ansible-check log -------------------------------------------- 2019-12-27 12:02:45 CST - TASK [ovirt-host-upgrade : Update system] ************************************** 2019-12-27 12:02:55 CST - fatal: [harrier....]: FAILED! => {"changed": false, "failures": [], "msg": "Unknown Error occured: Running QEMU processes found, cannot upgrade Vdsm.", "rc": 1, "results": []} 2019-12-27 12:02:55 CST - { "status" : "OK", "msg" : "", "data" : { "event" : "runner_on_failed", "uuid" : "74b495fa-2534-4850-b905-7255fed1fc56", "stdout" : "fatal: [harrier...]: FAILED! => {\"changed\": false, \"failures\": [], \"msg\": \"Unknown Error occured: Running QEMU processes found, cannot upgrade Vdsm.\", \"rc\": 1, \"results\": []}", "counter" : 26, "pid" : 27445, "created" : "2019-12-27T18:02:53.245099", "end_line" : 25, "runner_ident" : "094f2f72-28d3-11ea-bc88-525400060930", "start_line" : 24, "event_data" : { "play_pattern" : "all", "play" : "all", "event_loop" : null, "task_args" : "", "remote_addr" : "harrier....", "res" : { "_ansible_no_log" : false, "changed" : false, "results" : [ ], "msg" : "Unknown Error occured: Running QEMU processes found, cannot upgrade Vdsm.", "rc" : 1, "invocation" : { "module_args" : { "lock_timeout" : 300, "update_cache" : true, "conf_file" : null, "exclude" : [ ], "allow_downgrade" : false, "disable_gpg_check" : false, "disable_excludes" : null, "validate_certs" : true, "state" : "latest", "disablerepo" : [ ], "releasever" : null, "skip_broken" : false, "autoremove" : false, "download_dir" : null, "installroot" : "/", "install_weak_deps" : true, "name" : [ "*" ], "download_only" : false, "bugfix" : false, "list" : null, "install_repoquery" : true, "update_only" : false, "disable_plugin" : [ ], "enablerepo" : [ ], "security" : false, "enable_plugin" : [ ] } }, "failures" : [ ] }, "pid" : 27445, "play_uuid" : "52540006-0930-56fb-1424-000000000006", "task_uuid" : "52540006-0930-56fb-1424-000000000012", "task" : "Update system", "playbook_uuid" : "758d0326-a52a-4fe9-87b3-6be0d3492784", "playbook" : "ovirt-host-upgrade.yml", "task_action" : "yum", "host" : "harrier....", "ignore_errors" : null, "role" : "ovirt-host-upgrade", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-upgrade/tasks/main.yml:18" }, "parent_uuid" : "52540006-0930-56fb-1424-000000000012" } }

1 1

I had a crash yesterday in my ovirt cluster,I just tried to add a new network，Which migration Policy do you recommend?
by zhouhao＠vip.friendtimes.net 27 Dec '19

27 Dec '19

I had a crash yesterday in my ovirt cluster, which is made up of 3 nodes. I just tried to add a new network, but the whole cluster crashed I added a new network to my cluster, but while I was debugging the newswitch, when the switch was poweroff, the node detected the network card status down and then moved to Non-Operational state. At this time all of 3 nodes moved to Non-Operational state. All virtual machines have started automatic migration,When I received the alert email, all virtual machines were suspended In 15 minutes my newswitch were power up again.The 3 ovirt-nodes become active again, but many virtual machines become unresponsive or suspended due to forced migration, and only a few virtual machines are pulled up again due to cancelled migration After I tried to terminate the migration tasks and restart ovirt-engine service, I was still unable to restore most of the virtual machines, so I had to restart 3 ovirt-nodes to restore my virtual machine I didn't recover all the virtual machines until an hour later Then I modify my migration policy to " Do Not Migrate Virtual Machines" Which migration Policy do you recommend? I'm afraid to use cluster... zhouhao(a)vip.friendtimes.net

1 0

Re: Overt Node | Non-Responsive
by Strahil 26 Dec '19

26 Dec '19

Hi Vijay, Can you share the error messages in thelogs ,as the screenshot shows only 'INFO' messages. Best Regards, Strahil NikolovOn Dec 26, 2019 10:17, Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: > > Dear Team, > > > > I have a host on which I am using local storage for my VMs. Now I rebooted my host because it was shifted to another location, right now in vdsm logs I see errors like: > > > > > > Where in host I can see my mounted LVM volume is perfectly fine. I started VSDM services and libvirtd, but host is not leaving Non-Responsive state. > > > > Any help ..!! > > > > > > Vijay Sachdeva > > > >

1 0

Re: oVirt and Containers
by Jan Zmeskal 26 Dec '19

26 Dec '19

Okay, so this topic is quite vast, but I believe I can at the very least give you a few pointers and maybe others might chime in as well. Firstly, there's the Kubevirt project. It enables you to manage both application containers and virtual machines workloads (that cannot be easily containerized) in a shared environment. Another benefit is getting advantages of the powerful Kubernetes scheduler. I myself am not too familiar with Kubevirt, so I can only offer this high-level overview. More info here: https://kubevirt.io/ Then there is another approach which I am more familiar with. You might want to use oVirt as an infrastructure layer on top of which you run containerized workflow. This is achieved by deploying either OpenShift <https://www.openshift.com/> or the upstream project OKD <https://www.okd.io/> in the oVirt virtual machines. In that scenario, oVirt VMs are considered by OpenShift as compute resources and are used for scheduling containers. There are some advantages to this setup and two come into mind. Firstly, you can scale such OpenShift cluster up or down by adding/removing oVirt VMs according to your needs. Secondly, you don't need to set up all of this yourself. For OpenShift 3, Red Hat provides detailed guide on how to go about this. Part of that guide are Ansible playbooks that automate the deployment for you as long as you provide required variables. More info here: https://docs.openshift.com/container-platform/3.11/install_config/configuri… When it comes to OpenShift 4, there are two types of deployment. There's UPI - user provisioned infrastructure. In that scenario, you prepare all the resources for OpenShift 4 beforehand and deploy it in that existing environment. And there's also IPI - installer provisioned infrastructure. This means that you just give the installer access to your environment (e.g. AWS public cloud) and the installer provisions resources for itself based on recommendations and best practices. At this point, neither UPI nor IPI is supported for oVirt. However there is a GitHub repository <https://github.com/sa-ne/openshift4-rhv-upi> that can guide you through UPI installation on oVirt and also provides automation playbooks for that. I have personally followed the steps from the repository and deployed OpenShift 4.2 on top of oVirt without any major issues. As far as I remember, I might have needed occasional variable here and there but the process worked. Hope this helps! Jan On Tue, Dec 24, 2019 at 8:21 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > Hi Jan, > > Honestly, I didn't have anything specific in mind, just what is being used > out there today and what may be more prevalent. > > Just getting my oVIrt set up and want to know what might be recommended. > Would probably be mostly deploying images like Homeassistent, piHole, etc.. > for now. > > I guess if there is good oVirt direct integration, it would be nice to > keep it all in a single interface. > > Thanks.. > > ________________________________________ > From: Jan Zmeskal <jzmeskal(a)redhat.com> > Sent: Tuesday, December 24, 2019 1:54 PM > To: Robert Webb > Cc: users > Subject: Re: [ovirt-users] oVirt and Containers > > Hi Robert, > > there are different answers based on what you mean by integrating oVirt > and containers. Do you mean: > > - Installing container management (Kubernetes or OpenShift) on top of > oVirt and using oVirt as infrastructure? > - Managing containers from oVirt interface? > - Running VM workloads inside containers? > - Something different? > > I can elaborate more based on your specific needs > > Best regards > Jan > > On Tue, Dec 24, 2019 at 3:52 PM Robert Webb <rwebb(a)ropeguru.com<mailto: > rwebb(a)ropeguru.com>> wrote: > I was searching around to try and figure out the best way to integrate > oVirt and containers. > > I have found some sites that discuss it but all of them are like 2017 and > older. > > Any recommendations? > > Just build VM’s to host containers or is there some direct integration? > > Here are a couple of the old sites > > https://fromanirh.github.io/containers-in-ovirt.html > > > https://kubernetes.io/docs/setup/production-environment/on-premises-vm/ovir… > > > https://www.ovirt.org/develop/release-management/features/integration/conta… > _______________________________________________ > Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org<mailto: > users-leave(a)ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/7BIW2Z3ILBSAG… > > > -- > > Jan Zmeskal > > Quality Engineer, RHV Core System > > Red Hat <https://www.redhat.com> > > [ > https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9d… > ]<https://www.redhat.com> > > -- Jan Zmeskal Quality Engineer, RHV Core System Red Hat <https://www.redhat.com> <https://www.redhat.com>

2 1

Overt-Engine Down | Host Unresponsive
by Vijay Sachdeva 26 Dec '19

26 Dec '19

Dear Team, I have one question, if Ovirt-engine goes down for some reason. Will it impact my host to go un-responsive and all VMs down? I just faced a situation where my host was in un-responsive status and all VMs were down. I restart the ovirt-engine service and host got activated. Can anyone let me know this situation? Thanks Vijay Sachdeva

2 1

oVirt on a dedicated server with a single NIC and IP
by alexcustos＠gmail.com 26 Dec '19

26 Dec '19

Hi, My goal is to set up the latest oVirt on a dedicated server (Hetzner) with a single NIC and IP. Then using a Reverse Proxy and/or VPN (on the same host) to manage connections to VMs. I expected I could add a subnet as an alias to the main interface, or set it up on VLAN, or using vSwitch at the last resort, but all my attempts failed. Some configurations work until the first reboot, then VDSM screw up the main interface and I lost connection to the server, or Hosted Engine doesn't start from global maintenance. I've run out of good ideas already. I'm extremely sorry if this question already been discussed somewhere, but I found nothing helpful. I would be very appreciated for any hint on how to achieve this or at least a confirmation that the desired configuration is viable.

2 2

Overt Node | Non-Responsive
by Vijay Sachdeva 26 Dec '19

26 Dec '19

Dear Team, I have a host on which I am using local storage for my VMs. Now I rebooted my host because it was shifted to another location, right now in vdsm logs I see errors like: Where in host I can see my mounted LVM volume is perfectly fine. I started VSDM services and libvirtd, but host is not leaving Non-Responsive state. Any help ..!! Vijay Sachdeva

1 0

Re: [ovirt 4.2]vdsm host was shutdown unexpectly, configure of some vms was changed or lost when host was re-powered on and vm was started
by Strahil 25 Dec '19

25 Dec '19

Hi, If the VMs are starting properly and they don't use cloudinit - then the issue is not oVirt specific, but guest specific (Linux/Windows depending on guest OS). So your should check: 1. Does your host have any Netwprks out of sync (Host's network tab) If yes - put the server into maintenance and fix the issue (host's network tab) 2. Check each VM's configuration if it is defined to use CloudInit -> if yes, verify that cloudinit's service is running on the guest 3. Verify each problematic guest's network settings. If needed, set a static IP and try to ping another IP from the same subnet/Vlan . Best Regards, Strahil NikolovOn Dec 25, 2019 11:41, lifuqiong(a)sunyainfo.com wrote: >> >> >>> Dear All: >>> My ovirt engine managed two vdsms with nfs storage on another nfs server, it worked fine about three months. >>> One of host (host_1.3; ip:172.18.1.3) was created about 16 vms, but the host_1.3 was shutdown unexpectly about 2019-12-23 16:11, when re-started host and vms; half of the vms was lost some configure or changed , such as lost theirs ip etc.(the vm name is 'zzh_Chain49_ACG_M' in the vdsm.log) >>> >>> the vm zzh_Chain49_ACG_M was create by rest API through vm's templated . the template is 20.1.1.161; the vm zzh_Chain49_ACG_M was created by ovirt rest api and changed the ip to 20.1.1.219 >>> by ovirt rest api. But the ip was changed to templated ip when the accident happened. >>> >>> the vm's os is centos. >>> >>> Hope get help from you soon, Thank you. >>> >>> Mark >>> Sincerely.

1 0

[ovirt 4.2]vdsm host was shutdown unexpectly, configure of some vms was changed or lost when host was re-powered on and vm was started
by lifuqiong＠sunyainfo.com 25 Dec '19

25 Dec '19

Dear All: My ovirt engine managed two vdsms with nfs storage on another nfs server, it worked fine about three months. One of host (host_1.3; ip:172.18.1.3) was created about 16 vms, but the host_1.3 was shutdown unexpectly about 2019-12-23 16:11, when re-started host and vms; half of the vms was lost some configure or changed , such as lost theirs ip etc.(the vm name is 'zzh_Chain49_ACG_M' in the vdsm.log) the vm zzh_Chain49_ACG_M was create by rest API through vm's templated . the template is 20.1.1.161; the vm zzh_Chain49_ACG_M was created by ovirt rest api and changed the ip to 20.1.1.219 by ovirt rest api. But the ip was changed to templated ip when the accident happened. the vm's os is centos. Hope get help from you soon, Thank you. Mark Sincerely.

1 0

Re: Subscribe to mailing list and install oVirt with freeIPA
by Yedidyah Bar David 25 Dec '19

25 Dec '19

On Wed, Dec 25, 2019 at 10:45 AM Kien Pham <ptkcc(a)outlook.com> wrote: > > Dear oVirt team, > > I have post a thread on forum but I got problem with mailing list. How can I subscribe the mailing list. > > This is the question. > > Dear all, > > I am installing oVirt but I got an error when running engine-setup and get the following notification: > > [ INFO ] Restarting httpd > [ ERROR ] Failed to execute stage 'Closing up': Failed to start service 'httpd' > [ INFO ] Stage: Clean up > Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20191224111455-ppsefr.log > [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20191224111745-setup.conf' > [ INFO ] Stage: Pre-termination > [ INFO ] Stage: Termination > [ ERROR ] Execution of setup failed > > I think that the problem is freeIPA. I installed freeIPA to manage user authetication. But IPA is currently using port 80 so httpd service cannot start. Right. This isn't supported, for exactly this reason. The ovirt-engine rpm conflicts with ipa-server. How did you install it? I now see that there is also a package python2-ipaserver, and in fedora and el8 also python3-ipaserver. Now pushed a patch to conflict with them as well: https://gerrit.ovirt.org/105935 Anyway, you'll have to set them up on two different machines. Thanks for the report, and sorry for the failure... Good luck and best regards, -- Didi

1 0

Re: oVirt and Containers
by Jan Zmeskal 24 Dec '19

24 Dec '19

Hi Robert, there are different answers based on what you mean by integrating oVirt and containers. Do you mean: - Installing container management (Kubernetes or OpenShift) on top of oVirt and using oVirt as infrastructure? - Managing containers from oVirt interface? - Running VM workloads inside containers? - Something different? I can elaborate more based on your specific needs Best regards Jan On Tue, Dec 24, 2019 at 3:52 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > I was searching around to try and figure out the best way to integrate > oVirt and containers. > > > > I have found some sites that discuss it but all of them are like 2017 and > older. > > > > Any recommendations? > > > > Just build VM’s to host containers or is there some direct integration? > > > > Here are a couple of the old sites > > > > https://fromanirh.github.io/containers-in-ovirt.html > > > > > https://kubernetes.io/docs/setup/production-environment/on-premises-vm/ovir… > > > > > https://www.ovirt.org/develop/release-management/features/integration/conta… > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/7BIW2Z3ILBSAG… > -- Jan Zmeskal Quality Engineer, RHV Core System Red Hat <https://www.redhat.com> <https://www.redhat.com>

2 1

Subscribe to mailing list and install oVirt with freeIPA
by Kien Pham 24 Dec '19

24 Dec '19

Dear oVirt team, I have post a thread on forum but I got problem with mailing list. How can I subscribe the mailing list. This is the question. Dear all, I am installing oVirt but I got an error when running engine-setup and get the following notification: [ INFO ] Restarting httpd [ ERROR ] Failed to execute stage 'Closing up': Failed to start service 'httpd' [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20191224111455-ppsefr.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20191224111745-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed I think that the problem is freeIPA. I installed freeIPA to manage user authetication. But IPA is currently using port 80 so httpd service cannot start. Is there any suggest. Thank for all reply. Best regards. Kimloai.

1 0

Re: ovirt 4.3.7 + Gluster in hyperconverged (production design)
by Strahil 24 Dec '19

24 Dec '19

On Dec 24, 2019 02:18, Jayme <jaymef(a)gmail.com> wrote: > > If you can afford it I would definitely do raid. Being able to monitor and replace disks at the raid level is much easier than brick. With raid I’d do a gluster arbiter setup so your aren’t losing too much space. > > Keep an eye on libgfapi. It’s not default setting due to a few bugs but I’ve been testing it in my ssd hci cluster and have been seeing up to 5x io improvement. Also jumbo frames on those 10Gbps switches. > > Someone else will probably chime in re the other questions. I believe the GUI can only deploy a three sever cluster then you have to add the remaining hosts afterward. > > On Mon, Dec 23, 2019 at 5:56 PM <adrianquintero(a)gmail.com> wrote: >> >> Hi, >> After playing a bit with oVirt and Gluster in our pre-production environment for the last year, we have decided to move forward with a our production design using ovirt 4.3.7 + Gluster in a hyperconverged setup. >> >> For this we are looking get answers to a few questions that will help out with our design and eventually lead to our production deployment phase: >> >> Current HW specs (total servers = 18): >> 1.- Server type: DL380 GEN 9 >> 2.- Memory: 256GB >> 3.-Disk QTY per hypervisor: >> - 2x600GB SAS (RAID 0) for the OS I would pick Raid1 and split that between OS and brick for engine 's gluster volume >> - 9x1.2TB SSD (RAID[0, 6..]..? ) for GLUSTER. If you had bigger NICs I would preffer RAID0. Have you considered JBOD approach (each disk is presented as a standalone LUN from the raid array) ? Raid 6 is waste for your SSDs (when you already have replica 3 volumes). Ask HPE to provide SSDs from different manufacturing batches. I think that with SSD thee best compromise will be RAID 5 with replica3 for faster reads. As the SSDs will be overwritten with the same ammount of data, there could be a situation where all 3 from a replica set will be in predictive failiure (which is not nice) - that's why a raid 5 is a good option. >> 4.-Network: >> - Bond1: 2x10Gbps >> - Bond2: 2x10Gbps (for migration and gluster) Do not use Active-Backup bond as you will loose bandwidth. LACP (with hashing on layer 3+ layer4) is a good option, but it depends if your machines will be in the same LAN segment (communicating over layer 2 and no gateway inbetween). >> Our plan is to build 2x9 node clusters, however the following questions come up: >> >> 1.-Should we build 2 separate environments each with its own engine? or should we do 1 engine that will manage both clusters? If you decide to split - you have less fault tollerance - for example loosing 3 ot of 18 is better than loosing 3 out of 9 nodes :) >> 2.-What would be the best gluster volume layout for #1 above with regards to RAID configuration: >> - JBOD or RAID6 or…?. RAID6 wastes alot of space >> - what is the benefit or downside of using JBOD vs RAID 6 for this particular scenario? >> 3.-Would you recommend Ansible-based deployment (if supported)? If yes where would I find the documentation for it? or should we just deploy using the UI?. There is a topic in the mailing list about issues with ansible.It's up to you. >> - I have reviewed the following and in Chapter 5 it only mentions Web UI https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infras… >> - Also looked at https://github.com/gluster/gluster-ansible/tree/master/playbooks/hc-ansible… but could not get it to work properly. >> >> 4.-what is the recommended max server qty in a hyperconverged setup with gluster, 12, 15, 18...? I don't remmember but I think it was 9 nodes for Hyperconverged setup in RHV. >> Thanks, >> >> Adrian Best Regards, Strahil Nikolov

1 0

Re: Problems after 4.3.8 update
by Mahdi Adnan 24 Dec '19

24 Dec '19

We had an issue after upgrading Gluster nodes to 6.6, we had memory leak in gluster self heal daemon which cause it to do infinite heal and ate all of the nodes RAM "128GB", we had to downgrade to 6.5 to solve the problem. I think it is related to the changes made to the SHD code in version 6.6 ( 1760706 <https://bugzilla.redhat.com/1760706>) I might find the time and try to bug report the issue. Anyway, glad you solved your issue.

1 0

oVirt and Containers
by Robert Webb 24 Dec '19

24 Dec '19

I was searching around to try and figure out the best way to integrate oVirt and containers. I have found some sites that discuss it but all of them are like 2017 and older. Any recommendations? Just build VM's to host containers or is there some direct integration? Here are a couple of the old sites https://fromanirh.github.io/containers-in-ovirt.html https://kubernetes.io/docs/setup/production-environment/on-premises-vm/ovir… https://www.ovirt.org/develop/release-management/features/integration/conta…

1 0

Problems with ovirt-web-ui-1.6.0-1.el7.noarch
by Matthias Leopold 24 Dec '19

24 Dec '19

Hi, after upgrading oVirt from 4.3.5 to 4.3.7 some things in the VM Portal stopped working for users with "User" role: * running guest agent is not recognized ("It looks like no guest agent is configured on the VM.") * CD/ISO list is "[Empty]" for this I found "- limit CD/ISO list by data center" in rpm changelog, but this doesn't help me in any way Assigning other roles (UserVmRunTimeManager, PowerUserRole) doesn't change anything. /var/log/ovirt-engine/ui.log is quiet. Downgrading to ovirt-web-ui-1.5.3-1.el7.noarch resolves both issues. Am I the only one who is experiencing this? thx for advice Matthias

2 3

Re: Strategy to Mange Ovirt VMs Created from Templates
by Strahil 23 Dec '19

23 Dec '19

Sealing the VM is not something specific to oVirt, and is also valid for KVM. Actually sealing the VM is to run virt-sysprep against the VM/disk. You can check http://libguestfs.org/virt-sysprep.1.html and especially the --list-operations which can help you understand what ia being cleaned up. So you can: 1. Update the VM 2. Install and enable cloudinit service 3. Maybe try to create your ansible user and allow ssh keys (Using ansible over root is not good and is a bad practice) 4. Power off and seal the machine as a template (details at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualiza… ) 5. Create VM from the template and power it up manually. 6. Verify that the new VM matches your needs and remove. About cloud init - you should know that is used extensively in OpenStack and VmWare has it's own solution similar to CloudInit. Some examples can be found at: https://cloudinit.readthedocs.io/en/latest/topics/examples.html So you can change passwords, configure network, create your ansible user with necessary permissions and way more. Best Regards, Strahil NikolovOn Dec 23, 2019 18:46, jeremy_tourville(a)hotmail.com wrote: > > Thank you for your reply Luca, > In general your work flow is helpful and makes sense to me. > > I meant to say above- "As part of the template creation process ***the Ovirt docs*** say to seal the VM". > > So I think I understand that you need to use both processes (seal template + cloudinit) to get everything to work as desired. I'd still appreciate any more specifics about what sealing a VM does. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BRLNWOQB6FM4B…

1 0

Strategy to Mange Ovirt VMs Created from Templates
by jeremy_tourville＠hotmail.com 23 Dec '19

23 Dec '19

I want to be able to manage VMs using Ansible. As part of the template creation process it says to seal the VM. Can someone tell me what sealing does to a Linux VM? I understand it removes some of things that make the VM unique but no real specifics. So, if I want to manage a VM created from a template would this general process work?- Seal the VM Install CloudInit and keys, accounts, etc Shut off VM and create template from it. Create new VM using Ansbile & CloudInit CloudInit would have just enough info so that you could manage the VM with Ansible. Would that work? I am just starting to explore what CloudInit can do and what it is. I am brand new to it. I didn't find enough info on template sealing to help me devise a full cycle management strategy. Perhaps there are other/easier methods? Thanks for your advice and input.

3 4

[ovirt 4.2]vdsm host was shutdown unexpectly, configure of some vms was changed or lost when host was re-powered on and vm was started
by lifuqiong＠sunyainfo.com 23 Dec '19

23 Dec '19

Dear All: My ovirt engine managed two vdsms with nfs storage on another nfs server, it worked fine about three months. One of host (host_1.3; ip:172.18.1.3) was created about 16 vms, but the host_1.3 was shutdown unexpectly about 2019-12-23 16:11, when re-started host and vms; half of the vms was lost some configure or changed , such as lost theirs ip etc.(the vm name is 'zzh_Chain49_ACG_M' in the vdsm.log) the vm zzh_Chain49_ACG_M was create by rest API through vm's templated . the template is 20.1.1.161; the vm zzh_Chain49_ACG_M was created by ovirt rest api and changed the ip to 20.1.1.219 by ovirt rest api. But the ip was changed to templated ip when the accident happened. the vm's os is centos. Hope get help from you soon, Thank you. Mark Sincerely.

1 0

[ovirt 4.2]vdsm host was shutdown unexpectly, configure of some vms was changed or lost when host was re-powered on and vm was started
by lifuqiong＠sunyainfo.com 23 Dec '19

23 Dec '19

Dear All: My ovirt engine managed two vdsms with nfs storage on another nfs server, it worked fine about three months. One of host (host_1.3; ip:172.18.1.3) was created about 16 vms, but the host_1.3 was shutdown unexpectly about 2019-12-23 16:11, when re-started host and vms; half of the vms was lost some configure or changed , such as lost theirs ip etc.(the vm name is 'zzh_Chain49_ACG_M' in the vdsm.log) the vm zzh_Chain49_ACG_M was create by rest API through vm's templated . the template is 20.1.1.161; the vm zzh_Chain49_ACG_M was created by ovirt rest api and changed the ip to 20.1.1.219 by ovirt rest api. But the ip was changed to templated ip when the accident happened. the vm's os is centos. Hope get help from you soon, Thank you. Mark Sincerely.

1 0

VM Import Fails
by Vijay Sachdeva 23 Dec '19

23 Dec '19

Hi All, I am trying to import a VM from export domain, but import fails. Setup: Source DC has a NFS shared storage with two Hosts Destination DC has a local storage configured using LVM. Note: Used common export domain to export the VM. Anyone, please help me on this case to understand why it’s failing. Thanks Vijay Sachdeva

2 1

iSCSI connections and storage put into maintenance
by Gianluca Cecchi 22 Dec '19

22 Dec '19

Hello, I have a cluster composed by two hosts connected to some iSCSI storage domains. I have put one host into maintenance and I see that all the iSCSI sessions are closed. So far so good. Then I put one storage domain into maintenance and I would expect to see on the active host the (2) iSCSI sessions versus this storage domain to get closed. Instead I continue to see them up and "multipath -l" command gives me the 2 paths for the LUN part of the storage domain connection. Is this expected and only when I execute detach of the SD I will get the closure of the iSCSI connections? Because the detach phase will unregister the VMs and I would like not to do it, but at the same time I will have a planned maintenance for this storage domain and I would like to clean the connections on host side, to let multipath happy when they will go down for some hours... Thanks in advance for any insight. Gianluca

2 2

Ovirt 4.4 node installation failed for yum-utils
by usurse＠redhat.com 20 Dec '19

20 Dec '19

Has anyone got this issue? Host deploy failed with the requirement of yum-utils ~~~ 2019-12-19 20:16:54 IST - TASK [ovirt-host-deploy-facts : Install yum-utils] ***************************** 2019-12-19 20:17:27 IST - fatal: [Hostname: FAILED! => {"changed": false, "failures": ["No package yum-utils available."], "msg": "Failed to install some of the specified packages", "rc": 1, "results": []} 2019-12-19 20:17:27 IST - { "status" : "OK", "msg" : "", "data" : { "event" : "runner_on_failed", "uuid" : "9793bb18-562b-407e-9149-782b3e7f2d65", "stdout" : "fatal: [Hostname]: FAILED! => {\"changed\": false, \"failures\": [\"No package yum-utils available.\"], \"msg\": \"Failed to install some of the specified packages\", \"rc\": 1, \"results\": []}", ~~~

2 2

List VMS with ansible version 2.9 ovirt_vms_info module
by csyeow＠gmail.com 19 Dec '19

19 Dec '19

Hi All is someone use ansible to list guest vm and ovirt node on ansible 2.9 with ovirt_vms_info ? i read this https://lists.ovirt.org/pipermail/users/2017-May/081956.html but this is for ansible 2.8 and below with ovirt_vms_facts module. i new to ansible, hope someone manage to provide some sample playbook to start list vm on ovirt

3 5

[NEED YOUR INPUT] | Server Virtualization Trends 2019 - Statistical Survey | oVirt users
by Paweł Mączka 19 Dec '19

19 Dec '19

Hello everybody, Have you even wonder about worldwide virtualization market, who are undisputed leader and who is chasing the market from a back seat? I’m personally decided with my team (Storware) to work on data research on Server Virtualization Trends 2019. I'm contacting you because I believe, that you are using VM environment (for sure oVirt 😊) and you could give a value to our community. Your vote is crucial. My goal is to reach 1000 specialist (now we have 250 votes) on the market and convince them to vote 😊. I will be pleased if you agree to take part in this short survey (link below). The answer to the following questions will take you no more than 1 minute. Thank you in advance for your time, respondents' data will not be made public. I will inform you when the report is ready. http://bit.ly/VMTrends2019 Thanks In advance! Pozdrawiam/Best Regards Paweł Mączka Chief Technology Officer, VP +48 730 602 659 e-mail: p.maczka(a)storware.eu<mailto:p.maczka@storware.eu> Storware Sp. z o.o. ul. Leszno 8/44 01-192 Warszawa www.storware.eu<http://www.storware.eu/> [backupmoster (1)] Want to talk? Book a one-to-one session with me on https://calendly.com/p-maczka/30min/

1 0

Re: Strange disk behaviour within VMs
by Alessio Lustri 19 Dec '19

19 Dec '19

Hello, did you ever solved your issue? I’m experiencing the exact same thing!

1 0

vdsmd 4.4.0 throws an exception in asyncore.py while updating OVF data
by alexandermurashkin＠msn.com 19 Dec '19

19 Dec '19

vdsmd 4.4.0 throws an exception in asyncore.py while updating OVF data. First exception occured on December 14th and happens every hour since then. An hour before the first exception, we updated engine's and host's rpms packages. We did not run engine-backup so we cannot restore oVirt database if it got corrupted. Currently, both engine (RHEL 7.7) and host (RHEL 8.1) have the most recent packages (and were rebooted). Neither the updates, nor reboots resolve the issue. In general, oVirt is working. We can run virtual machines with disks in the storage domain. We can create new virtual machines. But some functionality is not available, for example, we cannot move disks between domains. The storage domain in question is on NFS. After the issue appeared, we successfully created Glusterfs domain but its OVF update also failed. Interesting that it was possible to move disks to it before OVF failure. Here is the vdsmd traceback for your convinience 2019-12-17 16:36:58,393-0600 ERROR (Reactor thread) [vds.dispatcher] uncaptured python exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher connected ('::ffff:172.20.1.142', 38002, 0, 0) at 0x7fbda865ed30> ( <class 'TypeError'>:object of type 'NoneType' has no len() [/usr/lib64/python3.6/asyncore.py|readwrite|108] [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_read|71] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py|handle_read|115] ) (betterAsyncore:179) I appreciate your help, Alexander Murashkin ------ Current Versions (that have the exception) ------ Engine ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch Host vdsm-4.40.0-1363.gitf6a1ba0a0.el8.x86_64 vdsm-python-4.40.0-1363.gitf6a1ba0a0.el8.noarch vdsm-yajsonrpc-4.40.0-1363.gitf6a1ba0a0.el8.noarch python3-libs-3.6.8-15.1.el8.x86_64 ------ December 13th Versions (that did not have the exception) ------ Engine ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch --- not sure, but probably the same as now Host vdsm-4.40.0-1360.git821afbbc2.el8.x86_64 vdsm-python-4.40.0-1360.git821afbbc2.el8.noarch vdsm-yajsonrpc-4.40.0-1360.git821afbbc2.el8.noarch python3-libs-3.6.8-15.1.el8.x86_64 ------ vdsm.log ------ 2019-12-17 16:36:58,393-0600 ERROR (Reactor thread) [vds.dispatcher] uncaptured python exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher connected ('::ffff:172.20.1.142', 38002, 0, 0) at 0x7fbda865ed30> (<class 'TypeError'>:object of type 'NoneType' has no len() [/usr/lib64/python3.6/asyncore.py|readwrite|108] [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_read|71] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py|handle_read|115]) (betterAsyncore:179) ----- engine.log ------ 2019-12-17 16:36:58,395-06 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [6c787bf3] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset (Failed with error VDS_NETWORK_ERROR and code 5022) ------ Web Interface Events ------ Storage Domain Events Dec 17, 2019, 4:36:58 PM Failed to update VMs/Templates OVF data for Storage Domain storedom3 in Data Center Default. 36b41d9c oVirt Dec 17, 2019, 4:36:58 PM Failed to update OVF disks fcc661df-b2e3-4625-be40-52b65033c6d7, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain storedom3). 36b41d9c oVirt Host Events Dec 17, 2019, 4:38:20 PM Status of host poplar was set to Up. 399a2181 oVirt Dec 17, 2019, 4:36:58 PM Host poplar is not responding. Host cannot be fenced automatically because power management for the host is disabled. oVirt

2 2

Re: Self hosted engine to hci
by Strahil 18 Dec '19

18 Dec '19

I have never done this , so you can simulate it on VMs before doint it : 1. Add the new host and put it in maintenance 2. You can backup your Hosted Engine VM and any critical VMs(just incase) 3. Create a new Gluster volume (either replica 3 or replica 2 arbiter 1) 4. Add the new volume via the UI as new storage 5. Do a storage migration from the distributed to the replica 3 volume. 6. Get rid of the old distributed volume and use it for something useful. 7. Remove the new hosts' maintenance and test live migration 8. Set global maintenance (via hosted-engine command) and poweroff the engine VM. 9. Manually power it up on a new host. 10. Remove global maintenance and shutdown a VM 11. Change the same VM to power up only on specific host and power it up. 12. Remove the limitation of that test VM (from step 10 & 11) And you are ready to go. Best Regards, Strahil Nikolov On Dec 18, 2019 16:39, Ernest Clyde Chua <ernestclydeachua(a)gmail.com> wrote: > > Hello currently i have a server running glusterfs on distributed 1 and a self hosted engine. > And planning to add two servers for hci. > > Do i add new hosts and change gluster volume type to replicated or backup all the vms then start from scratch? > > Is there any recommendation on this? > >

1 0

Self hosted engine to hci
by Ernest Clyde Chua 18 Dec '19

18 Dec '19

Hello currently i have a server running glusterfs on distributed 1 and a self hosted engine. And planning to add two servers for hci. Do i add new hosts and change gluster volume type to replicated or backup all the vms then start from scratch? Is there any recommendation on this?

1 0

NMA nodes
by suporte＠logicworks.pt 18 Dec '19

18 Dec '19

Hi, My host only shows one NUMA node. It means that I cannot setup a high performence VM? Thanks -- Jose Ferradeira http://www.logicworks.pt

2 3

ovirt hosted-engine on iSCSI offering one target
by wodel youchi 18 Dec '19

18 Dec '19

Hi, We have an oVirt Platforme using the 4.1 version. when the platforme was installed, it was made of : - Two HP Proliant DL380 G9 as hypervisors - One HP MSA1040 for iSCSI - One Synology for NFS - Two switches, one for network/vm traffic, the second for storage traffic. The problem : the hosted-engine domain was created using iSCSI on the HP MSA. The problem is that this disk array does not give the possibility to create different targets, it presents just one target. At that time we create both the hosted-engine and the first data domain using the same target, and we didn't pay attention to the information saying "i*f you are using iSCSI storage, do not use the same iSCSI target for the shared storage domain and data storage domain*". Question : - what problems can be generated by this (mis-)configuration? - Is this a must to do (correct) configuration. Regards.

3 2

Updating oVirt with gluster-ansible and Gluster 7?
by Victor Hooi 18 Dec '19

18 Dec '19

Hi, I'm planning to bring up a 3-node Ovirt/Gluster cluster, to do some comparison against an existing 3-node Proxmox/Ceph cluster we have. However, after reading the documentation Firstly, the oVirt documentation <https://www.ovirt.org/documentation/gluster-hyperconverged/chap-Single_node…> mentions that it uses gDeploy to setup Gluster. However, I saw from this commit <https://github.com/gluster/gdeploy/commit/57c19a4c233cccef0ccda468f697f1564…> that gDeploy has been deprecated since August 2018, in favour of gluster-ansible <https://github.com/gluster/gluster-ansible>. Is there a plan to move to gluster-ansible? Secondly, I saw there was a recently a new major Gluster release - Gluster 7 <https://www.gluster.org/announcing-gluster-7-0/>. Are there any plans to add this to oVirt? (I saw there was some performance improvements, hoping this will help with VM hosting). Thanks, Victor

1 0

Cannot start VM 2 hosts
by Stefan Wolf 17 Dec '19

17 Dec '19

I ve got 4 Hosts after changeing the harddrive on every Host and normal updates I am not able anymore to start VM's on 2 of these 4 hosts. I am also not able to migrate a running vm to these hosts. This is the error log for start up 2019-12-17 16:01:28,326+01 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[40cf6c27-6464-4fa7-bc01-9158cb03328b=VM]', sharedLocks=''}' 2019-12-17 16:01:28,609+01 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b'}), log id: 5706d380 2019-12-17 16:01:28,611+01 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5706d380 2019-12-17 16:01:28,854+01 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 40cf6c27-6464-4fa7-bc01-9158cb03328b Type: VMAction group RUN_VM with role type USER, ID: 40cf6c27-6464-4fa7-bc01-9158cb03328b Type: VMAction group EDIT_ADMIN_VM_PROPERTIES with role type ADMIN 2019-12-17 16:01:28,999+01 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{hostId='null', vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@147620b'}), log id: 612dabdf 2019-12-17 16:01:29,097+01 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] FINISH, UpdateVmDynamicDataVDSCommand, return: , log id: 612dabdf 2019-12-17 16:01:29,106+01 INFO [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] START, CreateVDSCommand( CreateVDSCommandParameters:{hostId='d38bae4c-8494-4861-ae5a-38992db338e5', vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b', vm='VM [z-push]'}), log id: 56abde11 2019-12-17 16:01:29,110+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] START, CreateBrokerVDSCommand(HostName = kvm360.durchhalten.intern, CreateVDSCommandParameters:{hostId='d38bae4c-8494-4861-ae5a-38992db338e5', vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b', vm='VM [z-push]'}), log id: 36d088ab 2019-12-17 16:01:29,163+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] VM <?xml version="1.0" encoding="UTF-8"?><domain type="kvm" xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> <name>z-push</name> <uuid>40cf6c27-6464-4fa7-bc01-9158cb03328b</uuid> <memory>4194304</memory> <currentMemory>4194304</currentMemory> <iothreads>1</iothreads> <maxMemory slots="16">16777216</maxMemory> <vcpu current="4">16</vcpu> <sysinfo type="smbios"> <system> <entry name="manufacturer">oVirt</entry> <entry name="product">OS-NAME:</entry> <entry name="version">OS-VERSION:</entry> <entry name="serial">HOST-SERIAL:</entry> <entry name="uuid">40cf6c27-6464-4fa7-bc01-9158cb03328b</entry> </system> </sysinfo> <clock offset="variable" adjustment="0"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> </clock> <features> <acpi/> </features> <cpu match="exact"> <model>SandyBridge</model> <feature name="pcid" policy="require"/> <feature name="spec-ctrl" policy="require"/> <feature name="ssbd" policy="require"/> <topology cores="1" threads="1" sockets="16"/> <numa> <cell id="0" cpus="0,1,2,3" memory="4194304"/> </numa> </cpu> <cputune/> <devices> <input type="mouse" bus="ps2"/> <channel type="unix"> <target type="virtio" name="ovirt-guest-agent.0"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/40cf6c27-6464-4fa7-bc01-9158cb03328b.ovirt-guest-agent.0"/> </channel> <channel type="unix"> <target type="virtio" name="org.qemu.guest_agent.0"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/40cf6c27-6464-4fa7-bc01-9158cb03328b.org.qemu.guest_agent.0"/> </channel> <rng model="virtio"> <backend model="random">/dev/urandom</backend> <alias name="ua-375334ff-0392-4314-879d-d89d81af5b69"/> </rng> <watchdog model="i6300esb" action="none"> <alias name="ua-5a4b6796-47bd-40e4-99ac-81b9546e6824"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x08" type="pci"/> </watchdog> <memballoon model="virtio"> <stats period="5"/> <alias name="ua-6d47e6f5-fd4f-452d-bcca-895b08d896f8"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x06" type="pci"/> </memballoon> <video> <model type="qxl" vram="32768" heads="1" ram="65536" vgamem="16384"/> <alias name="ua-79117281-c935-4cfc-aa21-9f028d817d79"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/> </video> <controller type="usb" model="piix3-uhci" index="0"> <address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"/> </controller> <controller type="virtio-serial" index="0" ports="16"> <alias name="ua-b82337cf-7f59-4390-ad5b-4c940e205fae"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x05" type="pci"/> </controller> <graphics type="spice" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" tlsPort="-1"> <channel name="main" mode="secure"/> <channel name="inputs" mode="secure"/> <channel name="cursor" mode="secure"/> <channel name="playback" mode="secure"/> <channel name="record" mode="secure"/> <channel name="display" mode="secure"/> <channel name="smartcard" mode="secure"/> <channel name="usbredir" mode="secure"/> <listen type="network" network="vdsm-ovirtmgmt"/> </graphics> <channel type="spicevmc"> <target type="virtio" name="com.redhat.spice.0"/> </channel> <interface type="bridge"> <model type="virtio"/> <link state="up"/> <source bridge="ovirtmgmt"/> <driver queues="4" name="vhost"/> <alias name="ua-db1017dd-9b76-4245-85fd-7490185434ac"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/> <boot order="2"/> <mac address="56:6f:c1:d7:00:02"/> <mtu size="1500"/> <filterref filter="vdsm-no-mac-spoofing"/> <bandwidth/> </interface> <disk type="file" device="cdrom" snapshot="no"> <driver name="qemu" type="raw" error_policy="report"/> <source file="" startupPolicy="optional"> <seclabel model="dac" type="none" relabel="no"/> </source> <target dev="hdc" bus="ide"/> <readonly/> <alias name="ua-c2fa60b6-be89-487c-b01f-ed14c2d4624f"/> <address bus="1" controller="0" unit="0" type="drive" target="0"/> </disk> <disk snapshot="no" type="file" device="disk"> <target dev="sda" bus="scsi"/> <source file="/rhev/data-center/0e279f02-0c73-11e9-bc47-00163e150480/80364cc8-8afd-4482-884d-d63d5a7988df/images/38f2e3bb-687f-4f01-b769-3a5403136181/1a9627d8-181b-4790-aa47-569dd1e6e9fb"> <seclabel model="dac" type="none" relabel="no"/> </source> <driver name="qemu" io="threads" type="qcow2" error_policy="stop" cache="none"/> <alias name="ua-38f2e3bb-687f-4f01-b769-3a5403136181"/> <address bus="0" controller="0" unit="0" type="drive" target="0"/> <boot order="1"/> <serial>38f2e3bb-687f-4f01-b769-3a5403136181</serial> </disk> </devices> <pm> <suspend-to-disk enabled="no"/> <suspend-to-mem enabled="no"/> </pm> <os> <type arch="x86_64" machine="pc-i440fx-rhel7.6.0">hvm</type> <smbios mode="sysinfo"/> <bootmenu enable="yes" timeout="30000"/> </os> <metadata> <ovirt-tune:qos/> <ovirt-vm:vm> <ovirt-vm:minGuaranteedMemoryMb type="int">4096</ovirt-vm:minGuaranteedMemoryMb> <ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion> <ovirt-vm:custom/> <ovirt-vm:device mac_address="56:6f:c1:d7:00:02"> <ovirt-vm:custom/> </ovirt-vm:device> <ovirt-vm:device devtype="disk" name="sda"> <ovirt-vm:poolID>0e279f02-0c73-11e9-bc47-00163e150480</ovirt-vm:poolID> <ovirt-vm:volumeID>1a9627d8-181b-4790-aa47-569dd1e6e9fb</ovirt-vm:volumeID> <ovirt-vm:imageID>38f2e3bb-687f-4f01-b769-3a5403136181</ovirt-vm:imageID> <ovirt-vm:domainID>80364cc8-8afd-4482-884d-d63d5a7988df</ovirt-vm:domainID> </ovirt-vm:device> <ovirt-vm:launchPaused>false</ovirt-vm:launchPaused> <ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior> </ovirt-vm:vm> </metadata> </domain> 2019-12-17 16:01:29,198+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] FINISH, CreateBrokerVDSCommand, return: , log id: 36d088ab 2019-12-17 16:01:29,252+01 INFO [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] FINISH, CreateVDSCommand, return: WaitForLaunch, log id: 56abde11 2019-12-17 16:01:29,252+01 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] Lock freed to object 'EngineLock:{exclusiveLocks='[40cf6c27-6464-4fa7-bc01-9158cb03328b=VM]', sharedLocks=''}' 2019-12-17 16:01:29,319+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-5) [3beff980-0f1a-461b-a6da-89c6f4dea6b3] EVENT_ID: USER_STARTED_VM(153), VM z-push was started by admin@internal-authz (Host: kvm360.durchhalten.intern). 2019-12-17 16:01:30,412+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (EE-ManagedThreadFactory-engineScheduled-Thread-56) [] Fetched 1 VMs from VDS 'd38bae4c-8494-4861-ae5a-38992db338e5' 2019-12-17 16:01:30,620+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-56) [] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), User <UNKNOWN> got disconnected from VM z-push. 2019-12-17 16:01:30,766+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-56) [] START, DumpXmlsVDSCommand(HostName = kvm360.durchhalten.intern, Params:{hostId='d38bae4c-8494-4861-ae5a-38992db338e5', vmIds='[40cf6c27-6464-4fa7-bc01-9158cb03328b]'}), log id: 4be2467a 2019-12-17 16:01:30,774+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DumpXmlsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-56) [] FINISH, DumpXmlsVDSCommand, return: {40cf6c27-6464-4fa7-bc01-9158cb03328b=<?xml version="1.0" encoding="UTF-8"?><domain type="kvm" xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"><name>z-push</name><uuid>40cf6c27-6464-4fa7-bc01-9158cb03328b</uuid><memory>4194304</memory><currentMemory>4194304</currentMemory><iothreads>1</iothreads><maxMemory slots="16">16777216</maxMemory><vcpu current="4">16</vcpu><sysinfo type="smbios"><system><entry name="manufacturer">oVirt</entry><entry name="product">OS-NAME:</entry><entry name="version">OS-VERSION:</entry><entry name="serial">HOST-SERIAL:</entry><entry name="uuid">40cf6c27-6464-4fa7-bc01-9158cb03328b</entry></system></sysinfo><clock offset="variable" adjustment="0"><timer name="rtc" tickpolicy="catchup"></timer><timer name="pit" tickpolicy="delay"></timer ><timer name="hpet" present="no"></timer></clock><features><acpi></acpi></features><cpu match="exact"><model>SandyBridge</model><feature name="pcid" policy="require"></feature><feature name="spec-ctrl" policy="require"></feature><feature name="ssbd" policy="require"></feature><topology cores="1" threads="1" sockets="16"></topology><numa><cell id="0" cpus="0,1,2,3" memory="4194304"></cell></numa></cpu><cputune></cputune><devices><input type="mouse" bus="ps2"></input><channel type="unix"><target type="virtio" name="ovirt-guest-agent.0"></target><source mode="bind" path="/var/lib/libvirt/qemu/channels/40cf6c27-6464-4fa7-bc01-9158cb03328b.ovirt-guest-agent.0"></source></channel><channel type="unix"><target type="virtio" name="org.qemu.guest_agent.0"></target><source mode="bind" path="/var/lib/libvirt/qemu/channels/40cf6c27-6464-4fa7-bc01-9158cb03328b.org.qemu.guest_agent.0"></source></channel><rng model="virtio"><backend model="random">/dev/urandom</backend><alias name="ua-375334ff-0392 -4314-879d-d89d81af5b69"></alias></rng><watchdog model="i6300esb" action="none"><alias name="ua-5a4b6796-47bd-40e4-99ac-81b9546e6824"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x08" type="pci"></address></watchdog><memballoon model="virtio"><stats period="5"></stats><alias name="ua-6d47e6f5-fd4f-452d-bcca-895b08d896f8"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x06" type="pci"></address></memballoon><video><model type="qxl" vram="32768" heads="1" ram="65536" vgamem="16384"></model><alias name="ua-79117281-c935-4cfc-aa21-9f028d817d79"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"></address></video><controller type="usb" model="piix3-uhci" index="0"><address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"></address></controller><controller type="virtio-serial" index="0" ports="16"><alias name="ua-b82337cf-7f59-4390-ad5b-4c940e205fae"></alias><address bus="0x00" domain="0x0000" function=" 0x0" slot="0x05" type="pci"></address></controller><graphics type="spice" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" tlsPort="-1"><channel name="main" mode="secure"></channel><channel name="inputs" mode="secure"></channel><channel name="cursor" mode="secure"></channel><channel name="playback" mode="secure"></channel><channel name="record" mode="secure"></channel><channel name="display" mode="secure"></channel><channel name="smartcard" mode="secure"></channel><channel name="usbredir" mode="secure"></channel><listen type="network" network="vdsm-ovirtmgmt"></listen></graphics><channel type="spicevmc"><target type="virtio" name="com.redhat.spice.0"></target></channel><interface type="bridge"><model type="virtio"></model><link state="up"></link><source bridge="ovirtmgmt"></source><driver queues="4" name="vhost"></driver><alias name="ua-db1017dd-9b76-4245-85fd-7490185434ac"></alias><address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci">< /address><boot order="2"></boot><mac address="56:6f:c1:d7:00:02"></mac><mtu size="1500"></mtu><filterref filter="vdsm-no-mac-spoofing"></filterref><bandwidth></bandwidth></interface><disk type="file" device="cdrom" snapshot="no"><driver name="qemu" type="raw" error_policy="report"></driver><source file="" startupPolicy="optional"><seclabel model="dac" type="none" relabel="no"></seclabel></source><target dev="hdc" bus="ide"></target><readonly></readonly><alias name="ua-c2fa60b6-be89-487c-b01f-ed14c2d4624f"></alias><address bus="1" controller="0" unit="0" type="drive" target="0"></address></disk><disk snapshot="no" type="file" device="disk"><target dev="sda" bus="scsi"></target><source file="/rhev/data-center/0e279f02-0c73-11e9-bc47-00163e150480/80364cc8-8afd-4482-884d-d63d5a7988df/images/38f2e3bb-687f-4f01-b769-3a5403136181/1a9627d8-181b-4790-aa47-569dd1e6e9fb"><seclabel model="dac" type="none" relabel="no"></seclabel></source><driver name="qemu" io="threads" type="qcow2" error_polic y="stop" cache="none"></driver><alias name="ua-38f2e3bb-687f-4f01-b769-3a5403136181"></alias><address bus="0" controller="0" unit="0" type="drive" target="0"></address><boot order="1"></boot><serial>38f2e3bb-687f-4f01-b769-3a5403136181</serial></disk></devices><pm><suspend-to-disk enabled="no"></suspend-to-disk><suspend-to-mem enabled="no"></suspend-to-mem></pm><os><type arch="x86_64" machine="pc-i440fx-rhel7.6.0">hvm</type><smbios mode="sysinfo"></smbios><bootmenu enable="yes" timeout="30000"></bootmenu></os><metadata><ovirt-tune:qos></ovirt-tune:qos><ovirt-vm:vm><ovirt-vm:minGuaranteedMemoryMb type="int">4096</ovirt-vm:minGuaranteedMemoryMb><ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion><ovirt-vm:custom></ovirt-vm:custom><ovirt-vm:device mac_address="56:6f:c1:d7:00:02"><ovirt-vm:custom></ovirt-vm:custom></ovirt-vm:device><ovirt-vm:device devtype="disk" name="sda"><ovirt-vm:poolID>0e279f02-0c73-11e9-bc47-00163e150480</ovirt-vm:poolID><ovirt-vm:volumeID>1a9627d8-181b-4790-aa4 7-569dd1e6e9fb</ovirt-vm:volumeID><ovirt-vm:imageID>38f2e3bb-687f-4f01-b769-3a5403136181</ovirt-vm:imageID><ovirt-vm:domainID>80364cc8-8afd-4482-884d-d63d5a7988df</ovirt-vm:domainID></ovirt-vm:device><ovirt-vm:launchPaused>false</ovirt-vm:launchPaused><ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior></ovirt-vm:vm></metadata></domain>}, log id: 4be2467a 2019-12-17 16:01:30,918+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-56) [] VM '40cf6c27-6464-4fa7-bc01-9158cb03328b' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='5f17a444-36e8-4854-a51b-f355aab2d8a8', vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b'}', device='virtio-scsi', type='CONTROLLER', specParams='[ioThreadId=1]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2019-12-17 16:01:32,762+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-12) [] VM '40cf6c27-6464-4fa7-bc01-9158cb03328b' was reported as Down on VDS 'd38bae4c-8494-4861-ae5a-38992db338e5'(kvm360.durchhalten.intern) 2019-12-17 16:01:32,764+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-12) [] START, DestroyVDSCommand(HostName = kvm360.durchhalten.intern, DestroyVmVDSCommandParameters:{hostId='d38bae4c-8494-4861-ae5a-38992db338e5', vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 3478fcba 2019-12-17 16:01:33,283+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-12) [] FINISH, DestroyVDSCommand, return: , log id: 3478fcba 2019-12-17 16:01:33,283+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-12) [] VM '40cf6c27-6464-4fa7-bc01-9158cb03328b'(z-push) moved from 'WaitForLaunch' --> 'Down' 2019-12-17 16:01:33,510+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM z-push is down with error. Exit message: The name org.fedoraproject.FirewallD1 was not provided by any .service files. 2019-12-17 16:01:33,512+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-12) [] add VM '40cf6c27-6464-4fa7-bc01-9158cb03328b'(z-push) to rerun treatment 2019-12-17 16:01:33,743+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-12) [] Rerun VM '40cf6c27-6464-4fa7-bc01-9158cb03328b'. Called from VDS 'kvm360.durchhalten.intern' 2019-12-17 16:01:33,877+01 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-753) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM z-push on Host kvm360.durchhalten.intern. 2019-12-17 16:01:34,057+01 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-753) [] Lock Acquired to object 'EngineLock:{exclusiveLocks='[40cf6c27-6464-4fa7-bc01-9158cb03328b=VM]', sharedLocks=''}' 2019-12-17 16:01:34,085+01 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-753) [] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='40cf6c27-6464-4fa7-bc01-9158cb03328b'}), log id: 4074b804 2019-12-17 16:01:34,086+01 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-753) [] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4074b804 2019-12-17 16:01:34,134+01 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-753) [] Validation of action 'RunVmOnce' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS 2019-12-17 16:01:34,135+01 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-753) [] Lock freed to object 'EngineLock:{exclusiveLocks='[40cf6c27-6464-4fa7-bc01-9158cb03328b=VM]', sharedLocks=''}' 2019-12-17 16:01:34,300+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-753) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM z-push (User: admin@internal-authz). 2019-12-17 16:01:34,314+01 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (EE-ManagedThreadFactory-engine-Thread-754) [2ecacb63] Running command: ProcessDownVmCommand internal: true. 2019-12-17 16:01:46,128+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [] Fetched 0 VMs from VDS 'd38bae4c-8494-4861-ae5a-38992db338e5' and this for migration 2019-12-17 16:05:42,411+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[46755b18-ac33-4e62-b1f0-a746bfeed59b=VM]', sharedLocks=''}' 2019-12-17 16:05:42,685+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: 46755b18-ac33-4e62-b1f0-a746bfeed59b Type: VMAction group MIGRATE_VM with role type USER 2019-12-17 16:05:43,043+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='7895de21-d174-471f-a0e5-340d59be1bb0', vmId='46755b18-ac33-4e62-b1f0-a746bfeed59b', srcHost='kvm380.durchhalten.intern', dstVdsId='d104ef2f-98a4-4a99-b28d-2725294f8626', dstHost='kvm320.durchhalten.intern:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='62', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {li mit=-1, action={name=abort, params=[]}}]]', dstQemu='192.168.200.231'}), log id: 57fb24a3 2019-12-17 16:05:43,048+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] START, MigrateBrokerVDSCommand(HostName = kvm380.durchhalten.intern, MigrateVDSCommandParameters:{hostId='7895de21-d174-471f-a0e5-340d59be1bb0', vmId='46755b18-ac33-4e62-b1f0-a746bfeed59b', srcHost='kvm380.durchhalten.intern', dstVdsId='d104ef2f-98a4-4a99-b28d-2725294f8626', dstHost='kvm320.durchhalten.intern:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='62', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400] }}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='192.168.200.231'}), log id: 6765eb33 2019-12-17 16:05:44,068+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] FINISH, MigrateBrokerVDSCommand, return: , log id: 6765eb33 2019-12-17 16:05:44,126+01 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 57fb24a3 2019-12-17 16:05:44,205+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-12) [ec2a5fc9-20e5-45f8-a38b-2a5dc96dadfe] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: TestVM, Source: kvm380.durchhalten.intern, Destination: kvm320.durchhalten.intern, User: admin@internal-authz). 2019-12-17 16:05:44,588+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-2) [] VM '46755b18-ac33-4e62-b1f0-a746bfeed59b' was reported as Down on VDS 'd104ef2f-98a4-4a99-b28d-2725294f8626'(kvm320.durchhalten.intern) 2019-12-17 16:05:44,591+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-2) [] START, DestroyVDSCommand(HostName = kvm320.durchhalten.intern, DestroyVmVDSCommandParameters:{hostId='d104ef2f-98a4-4a99-b28d-2725294f8626', vmId='46755b18-ac33-4e62-b1f0-a746bfeed59b', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1afbc395 2019-12-17 16:05:44,853+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-2) [] Failed to destroy VM '46755b18-ac33-4e62-b1f0-a746bfeed59b' because VM does not exist, ignoring 2019-12-17 16:05:44,854+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-2) [] FINISH, DestroyVDSCommand, return: , log id: 1afbc395 2019-12-17 16:05:44,854+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-2) [] VM '46755b18-ac33-4e62-b1f0-a746bfeed59b'(TestVM) was unexpectedly detected as 'Down' on VDS 'd104ef2f-98a4-4a99-b28d-2725294f8626'(kvm320.durchhalten.intern) (expected on '7895de21-d174-471f-a0e5-340d59be1bb0') 2019-12-17 16:05:44,854+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-2) [] Migration of VM 'TestVM' to host 'kvm320.durchhalten.intern' failed: VM destroyed during the startup. 2019-12-17 16:05:44,861+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [] VM '46755b18-ac33-4e62-b1f0-a746bfeed59b'(TestVM) moved from 'MigratingFrom' --> 'Up' 2019-12-17 16:05:44,862+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [] Adding VM '46755b18-ac33-4e62-b1f0-a746bfeed59b'(TestVM) to re-run list 2019-12-17 16:05:44,927+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-9) [] Rerun VM '46755b18-ac33-4e62-b1f0-a746bfeed59b'. Called from VDS 'kvm380.durchhalten.intern' 2019-12-17 16:05:44,944+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-879) [] START, MigrateStatusVDSCommand(HostName = kvm380.durchhalten.intern, MigrateStatusVDSCommandParameters:{hostId='7895de21-d174-471f-a0e5-340d59be1bb0', vmId='46755b18-ac33-4e62-b1f0-a746bfeed59b'}), log id: 127dcc70 2019-12-17 16:05:44,952+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-879) [] FINISH, MigrateStatusVDSCommand, return: , log id: 127dcc70 2019-12-17 16:05:45,228+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-879) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: TestVM, Source: kvm380.durchhalten.intern, Destination: kvm320.durchhalten.intern). 2019-12-17 16:05:45,329+01 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (EE-ManagedThreadFactory-engine-Thread-879) [] Lock freed to object 'EngineLock:{exclusiveLocks='[46755b18-ac33-4e62-b1f0-a746bfeed59b=VM]', sharedLocks=''}' maybe someone can give me a hint thx shb

1 1

Re: Ovirt OVN help needed
by Strahil 16 Dec '19

16 Dec '19

Hi Dominik, All, I've checked 'https://lists.ovirt.org/archives/list/users@ovirt.org/thread/W6U4XJHNMYMD3W…' and the user managed to clear up and start over. I have removed the ovn-external-provider from UI, but I forgot to copy the data from the fields. Do you know any refference guide (or any tips & tricks) for adding OVN ? Thanks in advance. Best Regards, Strahil NikolovOn Dec 12, 2019 20:49, Strahil <hunter86_bg(a)yahoo.com> wrote: > > Hi Dominik, > > Thanks for the reply. > > Sadly the openstack module is missing on the engine and I have to figure it out. > > Can't I just undeploy the ovn and then redeploy it back ? > > Best Regards, > Strahil Nikolov > > On Dec 12, 2019 09:32, Dominik Holler <dholler(a)redhat.com> wrote: >> >> The cleanest way to clean up is to remove all entities on the OpenStack Network API on ovirt-provider-ovn, e.g. by something like >> https://gist.github.com/dominikholler/19bcdc5f14f42ab5f069086fd2ff5e37#file… >> This should work, if not, please report a bug. >> >> To bypass the ovirt-provider-ovn, which is not recommended and might end in an inconsistent state, you could use ovn-nbctl . >> >> >> >> On Thu, Dec 12, 2019 at 3:33 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >>> >>> Hi Community, >>> >>> can someone hint me how to get rid of some ports? I just want to 'reset' my ovn setup. >>> >>> Here is what I have so far: >>> >>> [root@ovirt1 openvswitch]# ovs-vsctl list interface >>> _uuid : be89c214-10e4-4a97-a9eb-1b82bc433a24 >>> admin_state : up >>> bfd : {} >>> bfd_status : {} >>> cfm_fault : [] >>> cfm_fault_status : [] >>> cfm_flap_count : [] >>> cfm_health : [] >>> cfm_mpid : [] >>> cfm_remote_mpids : [] >>> cfm_remote_opstate : [] >>> duplex : [] >>> error : [] >>> external_ids : {} >>> ifindex : 35 >>> ingress_policing_burst: 0 >>> ingress_policing_rate: 0 >>> lacp_current : [] >>> link_resets : 0 >>> link_speed : [] >>> link_state : up >>> lldp : {} >>> mac : [] >>> mac_in_use : "7a:7d:1d:a7:43:1d" >>> mtu : [] >>> mtu_request : [] >>> name : "ovn-25cc77-0" >>> ofport : 6 >>> ofport_request : [] >>> options : {csum="true", key=flow, remote_ip="192.168.1.64"} >>> other_config : {} >>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0, tx_packets=0} >>> status : {tunnel_egress_iface=ovirtmgmt, tunnel_egress_iface_carrier=up} >>> type : geneve >>> >>> _uuid : ec6a6688-e5d6-4346-ac47-ece1b8379440 >>> admin_state : down >>> bfd : {} >>> bfd_status : {} >>> cfm_fault : [] >>> cfm_fault_status : [] >>> cfm_flap_count : [] >>> cfm_health : [] >>> cfm_mpid : [] >>> cfm_remote_mpids : [] >>> cfm_remote_opstate : [] >>> duplex : [] >>> error : [] >>> external_ids : {} >>> ifindex : 13 >>> ingress_policing_burst: 0 >>> ingress_policing_rate: 0 >>> lacp_current : [] >>> link_resets : 0 >>> link_speed : [] >>> link_state : down >>> lldp : {} >>> mac : [] >>> mac_in_use : "66:36:dd:63:dc:48" >>> mtu : 1500 >>> mtu_request : [] >>> name : br-int >>> ofport : 65534 >>> ofport_request : [] >>> options : {} >>> other_config : {} >>> statistics : {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0} >>> status : {driver_name=openvswitch} >>> type : internal >>> >>> _uuid : 1e511b4d-f7c2-499f-bd8c-07236e7bb7af >>> admin_state : up >>> bfd : {} >>> bfd_status : {} >>> cfm_fault : [] >>> cfm_fault_status : [] >>> cfm_flap_count : [] >>> cfm_health : [] >>> cfm_mpid : [] >>> cfm_remote_mpids : [] >>> cfm_remote_opstate : [] >>> duplex : [] >>> error : [] >>> external_ids : {} >>> ifindex : 35 >>> ingress_policing_burst: 0 >>> ingress_policing_rate: 0 >>> lacp_current : [] >>> link_resets : 0 >>> link_speed : [] >>> link_state : up >>> lldp : {} >>> mac : [] >>> mac_in_use : "1a:85:d1:d9:e2:a5" >>> mtu : [] >>> mtu_request : [] >>> name : "ovn-566849-0" >>> ofport : 5 >>> ofport_request : [] >>> options : {csum="true", key=flow, remote_ip="192.168.1.41"} >>> other_config : {} >>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0, tx_packets=0} >>> status : {tunnel_egress_iface=ovirtmgmt, tunnel_egress_iface_carrier=up} >>> type : geneve >>> >>> >>> When I try to remove a port - it never ends (just hanging): >>> >>> [root@ovirt1 openvswitch]# ovs-vsctl --dry-run del-port br-int ovn-25cc77-0 >>> In journal I see only this: >>> дек 12 04:13:57 ovirt1.localdomain ovs-vsctl[22030]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --dry-run del-port br-int ovn-25cc77-0 >>> >>> The stranger part to me is the log output: >>> >>> [root@ovirt1 openvswitch]# grep ovn-25cc77-0 /var/log/openvswitch/*.log >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:26:28.642Z|00032|bridge|INFO|bridge br-int: added interface ovn-25cc77-0 on port 14 >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:45:15.646Z|00113|bridge|INFO|bridge br-int: deleted interface ovn-25cc77-0 on port 14 >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:45:15.861Z|00116|bridge|INFO|bridge br-int: added interface ovn-25cc77-0 on port 2 >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:50:36.678Z|00118|bridge|INFO|bridge br-int: deleted interface ovn-25cc77-0 on port 2 >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:52:31.180Z|00121|bridge|INFO|bridge br-int: added interface ovn-25cc77-0 on port 3 >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:55:09.734Z|00125|bridge|INFO|bridge br-int: deleted interface ovn-25cc77-0 on port 3 >>> /var/log/openvswitch/ovs-vswitchd.log:2019-12-12T01:58:15.138Z|00127|bridge|INFO|bridge br-int: added interface ovn-25cc77-0 on port 6 >>> >>> I'm also attaching the verbose output of the dryrun. >>> >>> Thanks in advance. >>> >>> Best Regards, >>> Strahil Nikolov >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IG7YJPINVAF2B…

3 3

Cannot forward traffic through VXLAN
by k.betsis＠gmail.com 16 Dec '19

16 Dec '19

Hi all I have a VM network created with some hosts and I have included a vyos router acting as a Layer 2 extension to another hypervisor through VXLAN. I can see traffic reaching VMs from the other hypervisor to the ovirt hosted VMs. I can see traffic leaving the VMs hosted on the ovirt hypervisor. However, i do not see return traffic reaching the vyos VXLAN hosted on ovirt. I believe the VM network drops return traffic based on the destination MAC address. However, i have created the VM Network with security disabled. Can you please assist on how to troubleshoot?

2 9

Postgres stuck on 100% after Engine upgrade from 4.2.8 to 4.3.7
by Vrgotic, Marko 16 Dec '19

16 Dec '19

Hi oVirt, Since oVirt upgrade from 4.2.8 to 4.3.7, SHE postgres is running 100% CPU with 4 processes, related to ovirt_engine_history DB: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2617 postgres 20 0 372668 30876 26352 R 100.0 0.2 63:40.19 postgres: ovirt_engine_history ovirt_engine_history 127.0.0.1(50226) INSERT 3554 postgres 20 0 368680 15120 11716 R 100.0 0.1 56:45.72 postgres: ovirt_engine_history ovirt_engine_history 127.0.0.1(50448) BIND 2623 postgres 20 0 368068 17640 14896 R 99.3 0.1 42:35.36 postgres: ovirt_engine_history ovirt_engine_history 127.0.0.1(50228) BIND 2880 postgres 20 0 368680 20404 17004 R 99.0 0.1 65:13.96 postgres: ovirt_engine_history ovirt_engine_history 127.0.0.1(50252) BIND Output of select * from pg_stat_activity; is giving me following: 16401 | ovirt_engine_history | 2617 | 16386 | ovirt_engine_history | | 127.0.0.1 | | 50226 | 2019-12-09 13:37:07.779261+01 | 2019-12-09 13:37:09.555704+01 | 2019-12-09 13:39:04.915431+01 | 2019-12-09 13:39:04.915432+01 | | | active | 260332629 | 260332629 | INSERT INTO vm_disks_usage_samples_history (history_datetime,vm_id,disks_usage) VALUES ($1,$2,$3) | client backend 16401 | ovirt_engine_history | 2623 | 16386 | ovirt_engine_history | | 127.0.0.1 | | 50228 | 2019-12-09 13:37:07.820982+01 | 2019-12-09 14:00:00.042448+01 | 2019-12-09 14:00:00.997228+01 | 2019-12-09 14:00:00.997229+01 | | | active | 260333458 | 260332629 | SELECT +| client backend | | | | | | | | | | | | | | | | | | history_datetime, +| | | | | | | | | | | | | | | | | | | vm_id, +| | | | | | | | | | | | | | | | | | | disks_usage +| | | | | | | | | | | | | | | | | | | FROM vm_disks_usage_samples_history +| | | | | | | | | | | | | | | | | | | WHERE history_datetime >= '2019-12-09 07:00:00.000000+0100' +| | | | | | | | | | | | | | | | | | | AND history_datetime < '2019-12-09 08:00:00.000000+0100' +| | | | | | | | | | | | | | | | | | | ORDER BY history_id DESC, +| | | | | | | | | | | | | | | | | | | vm_id | 16401 | ovirt_engine_history | 2624 | 16386 | ovirt_engine_history | | 127.0.0.1 | | 50230 | 2019-12-09 13:37:07.826391+01 | | | 2019-12-09 13:37:07.835886+01 | Client | ClientRead | idle | | | | client backend 16401 | ovirt_engine_history | 2625 | 16386 | ovirt_engine_history | | 127.0.0.1 | | 50232 | 2019-12-09 13:37:07.847262+01 | | | 2019-12-09 13:37:07.850961+01 | Client | ClientRead | idle | | | | client backend 13214 | postgres | 6275 | 10 | postgres | psql | | | -1 | 2019-12-09 14:37:13.890996+01 | 2019-12-09 14:46:36.439887+01 | 2019-12-09 14:46:36.439887+01 | 2019-12-09 14:46:36.439897+01 | | | active | | 260332629 | select * from pg_stat_activity; | client backend [A screenshot of a computer Description automatically generated] Any ideas why this could be happening? Kindly awaiting your reply. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo

2 3

Re: Problems after 4.3.8 update
by Strahil 15 Dec '19

15 Dec '19

I didn't report it as nobody has mentioned it and I thought it's a onetime issue. I am,now, quite confident that is a bug. Are you using the gluster fuse mounts ( the ones in /rhev...) or libgfapi ? Can you open a case in the bugzilla.redhat.com ? Best Regards, Strahil NikolovOn Dec 15, 2019 13:16, Jayme <jaymef(a)gmail.com> wrote: > > I compared each file across my nodes and synced them. It seems to have resolved my issue. > > I wonder if there is a problem with 6.5 to 6.6 upgrade that is causing the problem? It’s strange that it seems to have happened to more than one person. I was also following proper upgrade procedure. > > > > On Sun, Dec 15, 2019 at 3:09 AM <hunter86_bg(a)yahoo.com> wrote: >> >> I don't know. I had the same issues when I migrated my gluster from v6.5 to 6.6 (currently running v7.0). >> Just get the newest file and rsync it to the rest of the bricks. It will solve the '?????? ??????' problem. >> >> Best Regards, >> Strahil Nikolov >> В неделя, 15 декември 2019 г., 3:49:27 ч. Гринуич+2, Jayme <jaymef(a)gmail.com> написа: >> >> >> on that page it says to check open bugs and the migration bug you mention does not appear to be on the list. Has it been resolved or is it just missing from this page? >> >> On Sat, Dec 14, 2019 at 7:53 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >>> >>> Nah... this is not gonna fix your issue and is unnecessary. >>> Just compare the data from all bricks ... most probably the 'Last Updated' is different and the gfid of the file is different. >>> Find the brick that has the most fresh data, and replace (move away as a backup and rsync) the file from last good copy to the other bricks. >>> You can also run a 'full heal'. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> В събота, 14 декември 2019 г., 21:18:44 ч. Гринуич+2, Jayme <jaymef(a)gmail.com> написа: >>> >>> >>> *Update* >>> >>> Situation has improved. All VMs and engine are running. I'm left right now with about 2 heal entries in each glusterfs storage volume that will not heal. >>> >>> In all cases each heal entry is related to an OVF_STORE image and the problem appears to be an issue with the gluster metadata for those ovf_store images. When I look at the files shown in gluster volume heal info output I'm seeing question marks on the meta files which indicates an attribute/gluster problem (even though there is no split-brain). And I get input/output error when attempting to do anything with the files. >>> >>> If I look at the files on each host in /gluster_bricks they all look fine. I only see question marks on the meta files when look at the file in /rhev mounts >>> >>> Does anyone know how I can correct the attributes on these OVF_STORE files? I've tried putting each host in maintenance and re-activating to re-mount gluster volumes. I've also stopped and started all gluster volumes. >>> >>> I'm thinking I might be able to solve this by shutting down all VMs and placing all hosts in maintenance and safely restarting the entire cluster.. but that may not be necessary? >>> >>> On Fri, Dec 13, 2019 at 12:59 AM Jayme <jaymef(a)gmail.com> wrote: >>>> >>>> I believe I was able to get past this by stopping the engine volume then unmounting the glusterfs engine mount on all hosts and re-starting the volume. I was able to start hostedengine on host0. >>>> >>>> I'm still facing a few problems: >>>> >>>> 1. I'm still seeing this issue in each host's logs: >>>> >>>> Dec 13 00:57:54 orchard0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': 'd70b171e-7488-4d52-8cad-bbc581dbf16e', 'volumeID': u'2632f423-ed89-43d9-93a9-36738420b866', 'imageID': u'd909dc74-5bbd-4e39-b9b5-755c167a6ee8'} failed:#012(code=201, message=Volume does not exist: (u'2632f423-ed89-43d9-93a9-36738420b866',)) >>>> Dec 13 00:57:54 orchard0 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs >>>> >>>> >>>> 2. Most of my gluster volumes still have un-healed entires which I can't seem to heal. I'm not sure what the answer is here. >>>> >>>> On Fri, Dec 13, 2019 at 12:33 AM Jayme <jaymef(a)gmail.com> wrote: >>>>> >>>>> I was able to get the hosted engine started manually via Virsh after re-creating a missing symlink in /var/run/vdsm/storage -- I later shut it down and am still having the same problem with ha broker starting. It appears that the problem *might* be with a corrupt ha metadata file, although gluster is not stating there is split brain on the engine volume >>>>> >>>>> I'm seeing this: >>>>> >>>>> ls -al /rhev/data-center/mnt/glusterSD/orchard0\:_engine/d70b171e-7488-4d52-8cad-bbc581dbf16e/ha_agent/ >>>>> ls: cannot access /rhev/data-center/mnt/glusterSD/orchard0:_engine/d70b171e-7488-4d52-8cad-bbc581dbf16e/ha_agent/hosted-engine.metadata: Input/output error >>>>> total 0 >>>>> drwxr-xr-x. 2 vdsm kvm 67 Dec 13 00:30 . >>>>> drwxr-xr-x. 6 vdsm kvm 64 Aug 6 2018 .. >>>>> lrwxrwxrwx. 1 vdsm kvm 132 Dec 13 00:30 hosted-engine.lockspace -> /var/run/vdsm/storage/d70b171e-7488-4d52-8cad-bbc581dbf16e/03a8ee8e-91f5-4e06-904b-9ed92a9706eb/db2699ce-6349-4020-b52d-8ab11d01e26d >>>>> l?????????? ? ? ? ? ? hosted-engine.metadata >>>>> >>>>> Clearly showing some sort of issue with hosted-engine.metadata on the client mount. >>>>> >>>>> on each node in /gluster_bricks I see this: >>>>> >>>>> # ls -al /gluster_bricks/engine/engine/d70b171e-7488-4d52-8cad-bbc581dbf16e/ha_agent/ >>>>> total 0 >>>>> drwxr-xr-x. 2 vdsm kvm 67 Dec 13 00:31 . >>>>> drwxr-xr-x. 6 vdsm kvm 64 Aug 6 2018 .. >>>>> lrwxrwxrwx. 2 vdsm kvm 132 Dec 13 00:31 hosted-engine.lockspace -> /var/run/vdsm/storage/d70b171e-7488-4d52-8cad-bbc581dbf16e/03a8ee8e-91f5-4e06-904b-9ed92a9706eb/db2699ce-6349-4020-b52d-8ab11d01e26d >>>>> lrwxrwxrwx. 2 vdsm kvm 132 Dec 12 16:30 hosted-engine.metadata -> /var/run/vdsm/storage/d70b171e-7488-4d52-8cad-bbc581dbf16e/66bf05fa-bf50-45ec-98d8-d00002040317/a2250415-5ff0-42ab-8071-cd9d67c3048c >>>>> >>>>> ls -al /var/run/vdsm/storage/d70b171e-7488-4d52-8cad-bbc581dbf16e/66bf05fa-bf50-45ec-98d8-d00002040317/a2250415-5ff0-42ab-8071-cd9d67c3048c >>>>> -rw-rw----. 1 vdsm kvm 1073741824 Dec 12 16:48 /var/run/vdsm/storage/d70b171e-7488-4d52-8cad-bbc581dbf16e/66bf05fa-bf50-45ec-98d8-d00002040317/a2250415-5ff0-42ab-8071-cd9d67c3048c >>>>> >>>>> >>>>> I'm not sure how to proceed at this point. Do I have data corruption, a gluster split-brain issue or something else? Maybe I just need to re-generate metadata for the hosted engine? >>>>> >>>>> On Thu, Dec 12, 2019 at 6:36 PM Jayme <jaymef(a)gmail.com> wrote: >>>>>> >>>>>> I'm running a three server HCI. Up and running on 4.3.7 with no problems. Today I updated to 4.3.8. Engine upgraded fine, rebooted. First host updated fine, rebooted and let all gluster volumes heal. Put second host in maintenance, upgraded successfully, rebooted. Waited for gluster volumes to heal for over an hour but the heal process was not completing. I tried restarting gluster services as well as the host with no success. >>>>>> >>>>>> I'm in a state right now where there are pending heals on almost all of my volumes. Nothing is reporting split-brain, but the heals are not completing. >>>>>> >>>>>> All vms are still currently running except hosted engine. Hosted engine was running but on the 2nd host I upgraded I was seeing errors such as: >>>>>> >>>>>> Dec 12 16:34:39 orchard2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': 'd70b171e-7488-4d52-8cad-bbc581dbf16e', 'volumeID': u'2632f423-ed89-43d9-93a9-36738420b866', 'imageID': u'd909dc74-5bbd-4e39-b9b5-755c167a6ee8'} failed:#012(code=201, message=Volume does not exist: (u'2632f423-ed89-43d9-93a9-36738420b866',)) >>>>>> >>>>>> I shut down the engine VM and attempted a manual heal on the engine volume. I cannot start the engine on any host now. I get: >>>>>> >>>>>> The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. >>>>>> >>>>>> I'm seeing ovirt-ha-agent crashing on all three nodes: >>>>>> >>>>>> Dec 12 18:30:48 orchard0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' >>>>>> Dec 12 18:30:48 orchard0 abrt-server: Duplicate: core backtrace >>>>>> Dec 12 18:30:48 orchard0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2019-03-14-21:02:52-44318 >>>>>> Dec 12 18:30:48 orchard0 abrt-server: Deleting problem directory Python-2019-12-12-18:30:48-23193 (dup of Python-2019-03-14-21:02:52-44318) >>>>>> Dec 12 18:30:49 orchard0 vdsm[6087]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? >>>>>> Dec 12 18:30:49 orchard0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE >>>>>> Dec 12 18:30:49 orchard0 systemd: Unit ovirt-ha-broker.service entered failed state. >>>>>> Dec 12 18:30:49 orchard0 systemd: ovirt-ha-broker.service failed. >>>>>> Dec 12 18:30:49 orchard0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. >>>>>> Dec 12 18:30:49 orchard0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. >>>>>> Dec 12 18:30:49 orchard0 systemd: Stopped oVirt Hosted Engine High Availability Communications Broker. >>>>>> >>>>>> >>>>>> Here is what gluster volume heal info on engine looks like, it's similar on other volumes as well (although more heals pending on some of those): >>>>>> >>>>>> gluster volume heal engine info >>>>>> Brick gluster0:/gluster_bricks/engine/engine >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/d909dc74-5bbd-4e39-b9b5-755c167a6ee8/2632f423-ed89-43d9-93a9-36738420b866.meta >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/053171e4-f782-42d7-9115-c602beb3c826/627b8f93-5373-48bb-bd20-a308a455e082.meta >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe.backup >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/dom_md/ids >>>>>> Status: Connected >>>>>> Number of entries: 4 >>>>>> >>>>>> Brick gluster1:/gluster_bricks/engine/engine >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/d909dc74-5bbd-4e39-b9b5-755c167a6ee8/2632f423-ed89-43d9-93a9-36738420b866.meta >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe.backup >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/053171e4-f782-42d7-9115-c602beb3c826/627b8f93-5373-48bb-bd20-a308a455e082.meta >>>>>> /d70b171e-7488-4d52-8cad-bbc581dbf16e/dom_md/ids >>>>>> Status: Connected >>>>>> Number of entries: 4 >>>>>> >>>>>> Brick gluster2:/gluster_bricks/engine/engine >>>>>> Status: Connected >>>>>> Number of entries: 0 >>>>>> >>>>>> I don't see much in vdsm.log and gluster logs look fairly normal to me, I'm not seeing any obvious errors in the gluster logs. >>>>>> >>>>>> As far as I can tell the underlying storage is fine. Why are my gluster volumes not healing and why is self-hosted engine failing to start? >>>>>> >>>>>> More agent and broker logs: >>>>>> >>>>>> ==> agent.log <== >>>>>> MainThread::ERROR::2019-12-12 18:36:09,056::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors >>>>>> MainThread::ERROR::2019-12-12 18:36:09,058::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent >>>>>> return action(he) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper >>>>>> return he.start_monitoring() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring >>>>>> self._initialize_broker() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker >>>>>> m.get('options', {})) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor >>>>>> ).format(t=type, o=options, e=e) >>>>>> RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] >>>>>> >>>>>> MainThread::ERROR::2019-12-12 18:36:09,058::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent >>>>>> MainThread::ERROR::2019-12-12 18:36:19,619::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors >>>>>> MainThread::ERROR::2019-12-12 18:36:19,619::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent >>>>>> return action(he) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper >>>>>> return he.start_monitoring() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring >>>>>> self._initialize_broker() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker >>>>>> m.get('options', {})) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor >>>>>> ).format(t=type, o=options, e=e) >>>>>> RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] >>>>>> >>>>>> MainThread::ERROR::2019-12-12 18:36:19,619::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent >>>>>> MainThread::ERROR::2019-12-12 18:36:30,568::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors >>>>>> MainThread::ERROR::2019-12-12 18:36:30,570::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent >>>>>> return action(he) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper >>>>>> return he.start_monitoring() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring >>>>>> self._initialize_broker() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker >>>>>> m.get('options', {})) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor >>>>>> ).format(t=type, o=options, e=e) >>>>>> RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] >>>>>> >>>>>> MainThread::ERROR::2019-12-12 18:36:30,570::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent >>>>>> MainThread::ERROR::2019-12-12 18:36:41,581::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors >>>>>> MainThread::ERROR::2019-12-12 18:36:41,583::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent >>>>>> return action(he) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper >>>>>> return he.start_monitoring() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring >>>>>> self._initialize_broker() >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker >>>>>> m.get('options', {})) >>>>>> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor >>>>>> ).format(t=type, o=options, e=e) >>>>>> RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] >>>>>> >>>>>> MainThread::ERROR::2019-12-12 18:36:41,583::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent >>>>>> >>>>>> >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/U5YFDWCQJYNAL…

2 1

Problems after 4.3.8 update
by Jayme 15 Dec '19

15 Dec '19

I'm running a three server HCI. Up and running on 4.3.7 with no problems. Today I updated to 4.3.8. Engine upgraded fine, rebooted. First host updated fine, rebooted and let all gluster volumes heal. Put second host in maintenance, upgraded successfully, rebooted. Waited for gluster volumes to heal for over an hour but the heal process was not completing. I tried restarting gluster services as well as the host with no success. I'm in a state right now where there are pending heals on almost all of my volumes. Nothing is reporting split-brain, but the heals are not completing. All vms are still currently running except hosted engine. Hosted engine was running but on the 2nd host I upgraded I was seeing errors such as: Dec 12 16:34:39 orchard2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed scanning for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID': '00000000-0000-0000-0000-000000000000', 'storagedomainID': 'd70b171e-7488-4d52-8cad-bbc581dbf16e', 'volumeID': u'2632f423-ed89-43d9-93a9-36738420b866', 'imageID': u'd909dc74-5bbd-4e39-b9b5-755c167a6ee8'} failed:#012(code=201, message=Volume does not exist: (u'2632f423-ed89-43d9-93a9-36738420b866',)) I shut down the engine VM and attempted a manual heal on the engine volume. I cannot start the engine on any host now. I get: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. I'm seeing ovirt-ha-agent crashing on all three nodes: Dec 12 18:30:48 orchard0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Dec 12 18:30:48 orchard0 abrt-server: Duplicate: core backtrace Dec 12 18:30:48 orchard0 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2019-03-14-21:02:52-44318 Dec 12 18:30:48 orchard0 abrt-server: Deleting problem directory Python-2019-12-12-18:30:48-23193 (dup of Python-2019-03-14-21:02:52-44318) Dec 12 18:30:49 orchard0 vdsm[6087]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Dec 12 18:30:49 orchard0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Dec 12 18:30:49 orchard0 systemd: Unit ovirt-ha-broker.service entered failed state. Dec 12 18:30:49 orchard0 systemd: ovirt-ha-broker.service failed. Dec 12 18:30:49 orchard0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Dec 12 18:30:49 orchard0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Dec 12 18:30:49 orchard0 systemd: Stopped oVirt Hosted Engine High Availability Communications Broker. Here is what gluster volume heal info on engine looks like, it's similar on other volumes as well (although more heals pending on some of those): gluster volume heal engine info Brick gluster0:/gluster_bricks/engine/engine /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/d909dc74-5bbd-4e39-b9b5-755c167a6ee8/2632f423-ed89-43d9-93a9-36738420b866.meta /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/053171e4-f782-42d7-9115-c602beb3c826/627b8f93-5373-48bb-bd20-a308a455e082.meta /d70b171e-7488-4d52-8cad-bbc581dbf16e/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe.backup /d70b171e-7488-4d52-8cad-bbc581dbf16e/dom_md/ids Status: Connected Number of entries: 4 Brick gluster1:/gluster_bricks/engine/engine /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/d909dc74-5bbd-4e39-b9b5-755c167a6ee8/2632f423-ed89-43d9-93a9-36738420b866.meta /d70b171e-7488-4d52-8cad-bbc581dbf16e/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe.backup /d70b171e-7488-4d52-8cad-bbc581dbf16e/images/053171e4-f782-42d7-9115-c602beb3c826/627b8f93-5373-48bb-bd20-a308a455e082.meta /d70b171e-7488-4d52-8cad-bbc581dbf16e/dom_md/ids Status: Connected Number of entries: 4 Brick gluster2:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 I don't see much in vdsm.log and gluster logs look fairly normal to me, I'm not seeing any obvious errors in the gluster logs. As far as I can tell the underlying storage is fine. Why are my gluster volumes not healing and why is self-hosted engine failing to start? More agent and broker logs: ==> agent.log <== MainThread::ERROR::2019-12-12 18:36:09,056::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2019-12-12 18:36:09,058::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] MainThread::ERROR::2019-12-12 18:36:09,058::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::ERROR::2019-12-12 18:36:19,619::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2019-12-12 18:36:19,619::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] MainThread::ERROR::2019-12-12 18:36:19,619::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::ERROR::2019-12-12 18:36:30,568::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2019-12-12 18:36:30,570::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] MainThread::ERROR::2019-12-12 18:36:30,570::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::ERROR::2019-12-12 18:36:41,581::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2019-12-12 18:36:41,583::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': '10.11.0.254'}] MainThread::ERROR::2019-12-12 18:36:41,583::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent

3 7

Re: Still having NFS issues. (Permissions)
by Shani Leviim 15 Dec '19

15 Dec '19

Hi Robert, I've found this one: https://www.ovirt.org/develop/troubleshooting-nfs-storage-issues.html IIRC, you don't need to use 'chown 36:36 /storage/', since no squash mode needed. Can you please share the result of 'cat /etc/exports'? It is supposed to be of the form: /storage *(rw,sync,no_root_squash) In addition, make sure the rpcbind and nfs-server services are running. Also, it seems there are a few mail threads about your issue, and it's hard to follow your steps and tries. Please avoid that by replying to this email instead of opening another thread. *Regards,* *Shani Leviim* On Sat, Dec 14, 2019 at 8:43 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > So I did some testing and and removed the “all_squash,anonuid=36,anongid=36”, > set all the image directories to 0755, added libvirt to the kvm group, then > rebooted. > > > > After doing so, sanlock had no access to the directories and neither did > libvert. Leaving everything else alone, I changed the the perms to 0760, > sanlock no longer complained, but libvirtd still complained about file > permissions. > > > > Next test was to the change file perms to 770 and I got the same error > with libvertd. > > > > I have not done any linux work for quite a while so please correct me, but > if I do a “ps aux | grep libvirt” I see the libvritd process running as > root. Does the libvirt user get invoked only when a script is running? If > the daemon is only running as root, then would it not be trying to access > storage as root at this point? > > > > This is my ps list: > > > > root 2898 0.1 0.0 1553860 28580 ? Ssl 14:45 0:01 > /usr/sbin/libvirtd –listen > > > > > > Here is what I see in the audit log: > > > > type=VIRT_CONTROL msg=audit(1576336098.295:451): pid=2898 uid=0 > auid=4294967295 ses=4294967295 > subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start > reason=booted vm="HostedEngine" uuid=70679ece-fbe9-4402-b9b0-34bbee9b6e69 > vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZHUU6CCXTIACJ… >

1 0

Re: Still having NFS issues. (Permissions)
by Robert Webb 14 Dec '19

14 Dec '19

It also appears that sanlock needs AT LEAST rw permissions on the group as rx breaks it per logs.

1 0

Re: Still having NFS issues. (Permissions)
by Robert Webb 14 Dec '19

14 Dec '19

So I did some testing and and removed the "all_squash,anonuid=36,anongid=36", set all the image directories to 0755, added libvirt to the kvm group, then rebooted. After doing so, sanlock had no access to the directories and neither did libvert. Leaving everything else alone, I changed the the perms to 0760, sanlock no longer complained, but libvirtd still complained about file permissions. Next test was to the change file perms to 770 and I got the same error with libvertd. I have not done any linux work for quite a while so please correct me, but if I do a "ps aux | grep libvirt" I see the libvritd process running as root. Does the libvirt user get invoked only when a script is running? If the daemon is only running as root, then would it not be trying to access storage as root at this point? This is my ps list: root 2898 0.1 0.0 1553860 28580 ? Ssl 14:45 0:01 /usr/sbin/libvirtd -listen Here is what I see in the audit log: type=VIRT_CONTROL msg=audit(1576336098.295:451): pid=2898 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="HostedEngine" uuid=70679ece-fbe9-4402-b9b0-34bbee9b6e69 vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed

1 0

Re: Unable to attach ISO domain to Datacenter
by Strahil 13 Dec '19

13 Dec '19

What is the output of 'ls -lZ /usr/bin/ionice' ? Here is mine : -rwxr-xr-x. root root system_u:object_r:bin_t:s0 /usr/bin/ionice Best Regards, Strahil NikolovOn Dec 13, 2019 16:58, Ivan Apolonio <ivan(a)apolonio.com.br> wrote: > > Hello. > > I tried to reinstall my whole environment using oficial oVirt ISO image and the same problem occurs. I also tried to give full access to vdsm user in sudoers file and the results are exactly the same. This is the error on showed on oVirt Manager: > > VDSM command AttachStorageDomainVDS failed: Cannot obtain lock: u"id=e65b4f0f-faa1-4cdc-9d2a-ab4fccfec0b0, rc=1, out=[], err=['setsid: failed to execute /usr/bin/ionice: Permission denied']" > > In other hand, the node server's vdsm.log file shows no errors. > > The engine.log still shows the same errors as my previous posts with logs. I don't know what else can I do. > > Please help. > Thanks, > Ivan > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/J6ZI7A7QTDZMV…

1 0

kernel parameter "spectre_v2=retpoline"
by Matthias Leopold 13 Dec '19

13 Dec '19

Hi, I'm planning to upgrade my installation from CentOS 7.6/oVirt 4.3.5 to CentOS 7.7/oVirt 4.3.7. According to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… there's a new default setting for Spectre V2 mitigation in new RHEL 7 installs. Shall I switch my hypervisor hosts to this setting when upgrading? Would newly installed CentOS 7 oVirt hosts have that kernel parameter? thx Matthias

2 2

Unable to attach ISO domain to Datacenter
by Ivan de Gusmão Apolonio 13 Dec '19

13 Dec '19

I'm having trouble to create a storage ISO Domain and attach it to a Datacenter. It just give me this error message: Error while executing action Attach Storage Domain: Could not obtain lock Also the oVirt's Engine log files show this error message: "setsid: failed to execute /usr/bin/ionice: Permission denied", but I was unable to identify what exactly it's trying to do to get this permission denied. 2019-11-14 16:46:07,779-03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-7388) [86161370-2aaa-4eff-9aab-c184bdf5bb98] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command AttachStorageDomainVDS failed: Cannot obtain lock: u"id=e6b34c42-0ca6-41f4-be3e-3c9b2af1747b, rc=1, out=[], err=['setsid: failed to execute /usr/bin/ionice: Permission denied']" This behavior just happens on ISO Domains, while Data Domains works fine. I have read oVirt documentation and searched everywhere but I was unable to find the solution for this issue. I'm using CentOS 7 with last update of all packages (oVirt version 4.3.6.7). Please help! Thanks, Ivan de Gusmão Apolonio

6 18