- Users - oVirt List Archives

Gluster FS & hosted engine fails to set up
by rob.downer＠orbitalsystems.co.uk 05 Dec '19

05 Dec '19

I have set up 3 new servers and as you can see Gluster is working well, however the hosted engine deployment fails can anyone suggest a reason ? I have wiped and set up all three servers again and set up Gluster first. This is the gluster congig I have used for the setup. Please review the configuration. Once you click the 'Finish Deployment' button, the management VM will be transferred to the configured storage and the configuration of your hosted engine cluster will be finalized. You will be able to use your hosted engine once this step finishes. * StorageStorage Type:glusterfs Storage Domain Connection:gfs3.gluster.private:/engine Mount Options:backup-volfile-servers=gfs2.gluster.private:gfs1.gluster.private Disk Size (GiB):58 [ INFO ] TASK [ovirt.hosted_engine_setup : Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Unexpected exception]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Unexpected exception]\". HTTP response code is 400."} root@ovirt3 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs3.gluster.private:/gluster_bricks/ data/data 49152 0 Y 3756 Brick gfs2.gluster.private:/gluster_bricks/ data/data 49153 0 Y 3181 Brick gfs1.gluster.private:/gluster_bricks/ data/data 49152 0 Y 15548 Self-heal Daemon on localhost N/A N/A Y 17602 Self-heal Daemon on gfs1.gluster.private N/A N/A Y 15706 Self-heal Daemon on gfs2.gluster.private N/A N/A Y 3348 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs3.gluster.private:/gluster_bricks/ engine/engine 49153 0 Y 3769 Brick gfs2.gluster.private:/gluster_bricks/ engine/engine 49154 0 Y 3194 Brick gfs1.gluster.private:/gluster_bricks/ engine/engine 49153 0 Y 15559 Self-heal Daemon on localhost N/A N/A Y 17602 Self-heal Daemon on gfs1.gluster.private N/A N/A Y 15706 Self-heal Daemon on gfs2.gluster.private N/A N/A Y 3348 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs3.gluster.private:/gluster_bricks/ vmstore/vmstore 49154 0 Y 3786 Brick gfs2.gluster.private:/gluster_bricks/ vmstore/vmstore 49152 0 Y 2901 Brick gfs1.gluster.private:/gluster_bricks/ vmstore/vmstore 49154 0 Y 15568 Self-heal Daemon on localhost N/A N/A Y 17602 Self-heal Daemon on gfs1.gluster.private N/A N/A Y 15706 Self-heal Daemon on gfs2.gluster.private N/A N/A Y 3348 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks

1 0

Re: Importing a Virtual Machine from a KVM Host
by Dirk Streubel 05 Dec '19

05 Dec '19

Hello Paul, thanks a lot for your answer. If i have any other questions during the import maybe i can asked again? And i let you know the result :) Regards Dirk Am 05.12.19 um 13:22 schrieb Staniforth, Paul: > Hello Dirk, > The link you are using is old developer docs, try the following. > https://www.ovirt.org/documentation/admin-guide/chap-External_Providers.html > > You need to setup an external provider and use a proxy host in the datacentre (not the engine host) you want to import into, this needs to be on the same network as your KVM host and have the virt-v2v package installed. > > Regards, > Paul S. > > -----Original Message----- > From: Dirk Streubel <dirk.streubel(a)posteo.de> > Sent: 05 December 2019 11:10 > To: users <users(a)ovirt.org> > Subject: [ovirt-users] Importing a Virtual Machine from a KVM Host > > Hello everbody, > > i am just a little bit confused about importing Qemu Images from my KVM Host. > > At home, and at work i am using a ovirt engine in a VM. The Version is 4.3.7 and on bare metal my host. > > Everything works fine but i have the same problem at home and on work. I want to import my qcow2 Images from a KVM Host but i > > will not work. > > The command virsh -r -c 'qemu+tcp://root@host1.example.org/system' list --all shows me akk my VMs. > > But when i want to load the images in the GUI, nothing happens. I follow this instruction: > > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > > After that, i take a look on the official 4.3 Rhev Documentation from Red Hat. There stand this: > > Log in to the proxy host and generate SSH keys for the vdsm user. > > sudo -u vdsm ssh-keygen > > So, i have no proxy host. So i want to do this on my ovirt-engine. Is this right? > > Next thing: the user vdsm is locked on the ovirt-engine machine and the host. > > So, do i have to unlocked the account of the user vdsm or what is the best way. > > My Problem is i found nothing that would help me to solve this problem. > > Maybe somebody here do me a favour and help me to solve this "challenge" > > Regards > > Dirk > > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/

1 0

Possible sources of cpu steal and countermeasures
by klaasdemter＠gmail.com 05 Dec '19

05 Dec '19

Hi, I'm having performance issues with a ovirt installation. It is showing high steal (5-10%) for a cpu intensive VM. The hypervisor however has more than 65% of his resources idle while the steal is seen inside of the VM. Even when placing only a single VM on a hypervisor it still receives steal (0-2%), even though the hypervisor is not overcommited. Hypervisor: 2 Socket system in total 2*28(56HT) cores VM: 30vCPUs (ovirt seems to think its a good idea to make that 15 sockets * 2 cores) My questions are: a) Could it be that the hypervisor is trying to schedule all 30 cores on a single numa node, ie using the HT cores instead of "real" ones and this shows up as steal? b) Do I need to make VMs this big numa-aware and spread the vm over both numa nodes? c) Would using the High Performance VM type help in this kind of situation? d) General advise: how do I reduce steal in an environment where the hypervisor has idle resources Any advise would be appreciated. Greetings Klaas

3 2

Re: Unable to attach ISO domain to Datacenter
by Strahil 05 Dec '19

05 Dec '19

Hi Ivan, Which file is this ? Contact me in private and I will send you my copy for comparison. Best Regards, Strahil NikolovOn Dec 4, 2019 18:55, Ivan Apolonio <ivan(a)apolonio.com.br> wrote: > > I've tried to commenting out this line on /etc/pam.d/password-auth: > > auth requisite pam_succeed_if.so uid >= 1000 quiet_success > > However, although it has stopped complaining about "requirement uid >= 1000 not met by user vdsm" on /var/log/secure , the error trying to attach ISO Domain to Datacenter still persists, showing the same error message, so it looks that this was not the cause of the problem :-( > > Going back to stage zero. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TTD4PH2FIUK23…

1 0

Ovirt-OSN integration
by siovelrm＠gmail.com 04 Dec '19

04 Dec '19

Hi, I want to use External Logical Networks Openstack Neutron. For this it is necessary to have all Openstack installed or is it possible to have only Neutron and used as a service? Sorry if the question is very obvious but I am starting now with this topic.

2 2

Re: 3node HCI fails when HostedEngineLocal is trying to add additional Gluster members
by Strahil 04 Dec '19

04 Dec '19

If you manage to get your VM on the gluster - you are almost done. I had similar situation and using virsh/hosted-engine , I have managed to reach the GUI. From there we can have a Clue what is going on. Usually there is a dependency: - Master storage domain should be UP -This allows the DC to become UP -Then a host in the cluster can become SPM (control storage) -And only then you can start your other gluster volumes. So I can only recommend you to try to reach GUI and check the situation. Then it will be moderately easy to bring everything up. Best Regards, Strahil NikolovOn Dec 4, 2019 17:06, thomas(a)hoberg.net wrote: > > Thanks Strahil, for your suggestions. > > Actually, I was far beyond the pick-up point you describe, as the Gluster had all been prepared and was operable, even the local VM was already running and accessible via the GUI. > > But I picked up your hint to try to continue with the scripted variant, and found that it allowed me much better insight into what was going on. > > I am a little worried, though, that it actually works a little differently from the GUI-wizard variant, in any case the failures don't seem identical. > That would have implications in terms of test-automation, which I'd rather not have to worry about. > > In a separate test for the same operation on a separate set of hardware the installation got significantly further, up to the point where the local VM had actually been moved onto the Gluster storage and into the Cluster, but then failed a validation step at the very end (while the VM is actually up and running, albeit only withe primary host and that listed as "Unresponsive" while it's hosting the VM....) > > I have opened a separate tickt for that... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FTCFIP7Q5KWXT…

1 0

Cannot activate/deactivate storage domain
by Albl, Oliver 04 Dec '19

04 Dec '19

Hi all, I run an oVirt 4.3.6.7-1.el7 installation (50+ hosts, 40+ FC storage domains on two all-flash arrays) and experienced a problem accessing single storage domains. As a result, hosts were taken "not operational" because they could not see all storage domains, SPM started to move around the hosts. oVirt messages start with: 2019-11-04 15:10:22.739+01 | VDSM HOST082 command SpmStatusVDS failed: (-202, 'Sanlock resource read failure', 'IO timeout') 2019-11-04 15:10:22.781+01 | Invalid status on Data Center <name>. Setting Data Center status to Non Responsive (On host HOST82, Error: General Exception). ... 2019-11-04 15:13:58.836+01 | Host HOST017 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.85+01 | Host HOST005 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.85+01 | Host HOST012 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.851+01 | Host HOST002 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.851+01 | Host HOST010 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.851+01 | Host HOST011 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:58.852+01 | Host HOST004 cannot access the Storage Domain(s) HOST_LUN_204 attached to the Data Center <name>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.011+01 | Host HOST017 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.238+01 | Host HOST004 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.249+01 | Host HOST005 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.255+01 | Host HOST012 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.273+01 | Host HOST002 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.279+01 | Host HOST010 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:13:59.386+01 | Host HOST011 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center <UNKNOWN>. Setting Host state to Non-Operational. 2019-11-04 15:15:14.145+01 | Storage domain HOST_LUN_221 experienced a high latency of 9.60953 seconds from host HOST038. This may cause performance and functional issues. Please consult your Storage Administrator. The problem mainly affected two storage domains (on the same array) but I also saw single messages for other storage domains (one the other array as well). Storage domains stayed available to the hosts, all VMs continued to run. When constantly reading from the storage domains (/bin/dd iflag=direct if=<metadata> bs=4096 count=1 of=/dev/null) we got expected 20+ MBytes/s on all but some storage domains. One of them showed "transfer rates" around 200 Bytes/s, but went up to normal performance from time to time. Transfer rate to this domain was different between the hosts. /var/log/messages contain qla2xxx abort messages on almost all hosts. There are no errors on SAN switches or storage array (but vendor is still investigating). I did not see high load on the storage array. The system seemed to stabilize when I stopped all VMs on the affected storage domain and this storage domain became "inactive". Currently, this storage domain still is inactive and we cannot place it in maintenance mode ("Failed to deactivate Storage Domain") nor activate it. OVF Metadata seems to be corrupt as well (failed to update OVF disks <id>, OVF data isn't updated on those OVF stores). The first six 512 byte blocks of /dev/<id>/metadata seem to contain only zeros. Any advice on how to proceed here? Is there a way to recover this storage domain? All the best, Oliver

4 13

3node HCI fails when HostedEngineLocal is trying to add additional Gluster members
by thomas＠hoberg.net 04 Dec '19

04 Dec '19

Three Gen8 HP360 recalled from retirement with single 1TB TLC SATA SSD for boot and oVirt /engine and 7x4TB HDD RAID6 for /vmstore and /data, 10Gbit NICs and network. All CentOS 7.7 updated daily. These machines may not be used exclusively for oVirt so I don't want to re-install the OS, when an oVirt setup fails: Instead I try my best to clean up the nodes when doing another oVirt installation run. They ran oVirt for a week or two using a completely distinct set of storage, so they are fundamentally sound, but we wanted higher storage capacity so I swapped everything and re-installed CentOS very much the same way as before. The first oVirt setup went smoothly but the cluster crumbled without much usage. I won't go into details here, because I didn't want to investigate for now, instead I focussed on redoing the installation and cleaning up the old setup. I know the docs actually recommend starting with wiped hardware, but operationally that would be a show-stopper for the intended use case. So I cleaned up the best I can (ovirt-hosted-engine-cleanup with and without redoing the whole Gluster storage setup, where apart from SSD caching not working, I don't have issues). Undoing the network changes in such a way that the oVirt HCI wizard ceases complaining is a bit more involved. I typically run: - vdsm-tool ovn-unconfigure - vdsm-tool clear-nets (now need to switch to the console) - vdsm-tool remove-config and then I still need to edit /etc/sysconfig/network-scripts/ifcfg-<ethernet-device> to bring the physical adapter back to life. Sometimes I still need to remove the ovirtmgmnt bridge manually etc. Whether I remove and redo the Gluster as a bit of an effect in re-installation, but it doesn't make a difference in what follows. So here is where I am currently getting stuck consistently: The wizard is gone through preparing the Gluster storage (which is completely functional at that point), has created the local VM on the installation node, installed the Postgres database, filled it etc. basically has oVirt up and running with the primary Gluster node and now would like to add the second and third nodes. At that point I get "Connection lost" in the Web-Wizard, evidently as a consequence of Ansible fiddling around heavily to set up the local bridge for the VM. I remember that for the scripted variant of the setup it is recommended to run the script behind 'screen' or 'tmux' in order to ensure its execution isn't interrupted by that. But for the GUI variant, evidently there *should* be some other type of potection, perhaps via the re-connecting nature of HTTP... Pushing the "Reconnect" button on the GUI at that point doesn't return you to the point of the setup, but only offers to redeploy, while the HostedEngineLocal is still there and running. I ssh'd into the machine and started looking for errors and warning and saw that the installation had gone rather far without incidence. OTOPI had completely finished the WildFly server is up and running the Postgres database fully installed and running smoothly, the only thing I can find is that it's trying to add the additional gluster nodes, but complains that these nodes (quotes gluster-UUIDs) are not part of the "cluster". An investigation into the Postgres database shows, that the 'gluster_server' table indeed only has the primary node in it. I don't know what part of the process should have added the other two nodes, but there seems to be no *remaining* connectivity issue with the Gluster members. I installed gscli and connected to all three nodes and volumes without issue. I am guessing at this point, that the complex rewiring of the software defined network is causing a temporary issue and a race condition that I don't know how to recover from. Since the oVirt management GUI is actually fully operational and can be reached from the primary node via the temporary bridge, I went into the GUI and even managed to add the additional two nodes without any problems. Their installation went through without any issues, they showed up in the gluster_servers table on Postgress and basically the installation could have proceeded from that point, except... that I don't know how to restart the process from that point: It still has to 'beam' the local VM into the Gluster storage and restart it there. I have gone through the process three times now, with absolutely identical results. I could use some help how to recover from that situation, which looks like a race condition, nothing a re-installation of everything would really resolve. In the mean-time, I'll try the scripted variant on 'screen' to see if that fares better.

2 3

Re: Certificate of host is invalid
by Strahil 04 Dec '19

04 Dec '19

Hi , You can try with: 1. Set the host in maintenance 2. From Install dropdown , select 'reinstall' and then configure the necessary info + whether you would like to use the host as Host for the HostedEngine VM. Once the reinstall (of Ovirt software) is OK, the node will be activated automatically. Best Regards, Strahil NikolovOn Nov 27, 2019 18:01, Jon bae <jonbae77(a)gmail.com> wrote: > > Hello everybody, > since last update to 4.3.7 I get this error message: > > Certificate of host host.name is invalid. The certificate doesn't contain valid subject alternative name, please enroll new certificate for the host. > > Have you an idea of how I can fix that? > > Regards > Jonathan

3 3

Re: Scenario: Ovirt-engine on hardware and recovery
by Yedidyah Bar David 04 Dec '19

04 Dec '19

On Tue, Dec 3, 2019 at 3:44 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > > Didi, > > Thanks for the reply. > > Yes, backups are done, I am just thinking in the case of Murphy's Law where nothing works and the engine is gone. I am also new to oVirt and am always loking for the "proper" backup method as you have laready mentioned using engine-backup. > > Storage is NFS backed and backups are done in case of storage server failure. OK. I suggest, then, that you methodically plan, document and test your environment on various different failure modes. Since you are new to oVirt, it's obvious you still do not have the full mental image of all the components, their behaviour, etc., and even more so how they might react, or be handled, in emergency situations. In particular, spend some time learning about the SPM, master storage domain, HA VMs and how they might be affected by/affecting restore flows, etc. Good luck! > > What is the preferred VM backup method so both disk and VM config is captured? Would rather not have to put agents on all the VM for OS level backups and then have to manually recreate all of the machines. Do all the VM configs get stored on the shared storage just like one of the other virtualization vendors? oVirt does not supply, nor endorse, a specific "preferred method" for this. Rather, it exposes an API, and provides a simple example script: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/vm_backu… There are several backup products/projects, both large and small, both open and closed source, that use this API and can be used. I personally never used any, so can't recommend anything. If you search a bit the list archives you'll find at least some of them. Best regards, > > Appreciate the time from folks on the list who respond. > ________________________________ > From: Yedidyah Bar David <didi(a)redhat.com> > Sent: Tuesday, December 3, 2019 3:29 AM > To: Robert Webb <rwebb(a)ropeguru.com> > Cc: users <users(a)ovirt.org> > Subject: Re: [ovirt-users] Scenario: Ovirt-engine on hardware and recovery > > On Tue, Dec 3, 2019 at 4:38 AM <rwebb(a)ropeguru.com> wrote: > > > > So thinking about my setup, I am thinking through different failure scenarios. > > > > So lets say I have a small physical server with 8 cores and 16GB RAM and I install Centos 7 and ovirt-engine on bare metal. This would also be the same scenario is the engine were on a VM. > > > > I run into a major hardware issue and completely lose the engine. > > You do take backups, right? :-) > > For the engine itself, you can use engine-backup. > > > > > How does one recover the cluster setup and not have to start from scratch by having to rebuild all the nodes? Can the engine just be rebuilt and the ovirt nodes be imported? > > You didn't mention your plans re storage. > > If you put your VMs on well-handled storage, and lose everything but > the storage, you can install a new engine, new nodes, and then import > the VMs from the existing storage. You will still miss some > configuration etc., but should mostly be ok. > > If you plan to use gluster storage using the same nodes, aka HCI, the > same applies, obviously, only that the nodes are your storage, and not > an external system. You still have to make sure you handle them > properly - backup, DR, etc., depending on your needs and budget. > > If you do have engine-backup backups, but lost all your storage, > engine-backup will not help you. > > > > > > This scenario is based on the nodes being built from the ovirt node iso. > > That's ok, but is not relevant. Main question is the storage. Can be > external or local/gluster with either ovirt-node or EL. > > Good luck, > > -- > Didi > -- Didi

1 0