- Users - oVirt List Archives

Re: self-hosted ovirt-engine down
by Strahil 18 Mar '19

18 Mar '19

In order to be easier to read, you can rotate the log before starting the engine.Should be something like: hosted-engine --set-maintenance --mode=global && logrotate -f /etc/vdsm/logrotate/vdsm && hosted-engine -- vm-start Best Regards, Strahil NikolovOn Mar 16, 2019 10:52, Simone Tiraboschi <stirabos(a)redhat.com> wrote: > > > > On Sat, Mar 16, 2019 at 7:44 AM <siovelrm(a)gmail.com> wrote: >> >> Hi, I have a big problem with ovirt. I use version 4.2.7 with self-hosted. The problem is that when I try to raise the vm of the ovirt-engine with the command: hosted-engine --vm-start, it appears in the output >> "VM exists and is down, cleaning up and restarting" >> when running: hosted-engine --vm-status appears >> --== Host 1 status ==-- >> >> conf_on_shared_storage : True >> Status up-to-date : True >> Hostname : node1.softel.cu >> Host ID : 1 >> Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down_unexpected", "detail": "Down"} >> Score : 0 >> stopped : False >> Local maintenance : False >> crc32 : 02c3b5a4 >> local_conf_timestamp : 49529 >> Host timestamp : 49529 >> Extra metadata (valid at timestamp): >> metadata_parse_version=1 >> metadata_feature_version=1 >> timestamp=49529 (Sat Mar 16 02:39:10 2019) >> host-id=1 >> score=0 >> vm_conf_refresh_time=49529 (Sat Mar 16 02:39:11 2019) >> conf_on_shared_storage=True >> maintenance=False >> state=EngineUnexpectedlyDown >> stopped=False >> timeout=Thu Jan 1 08:49:39 1970 >> >> in /var/log/messages >> Mar 16 02:35:34 node1 vdsm[26151]: WARN Attempting to remove a non existing network: ovirtmgmt/0c3e1c08-3928-47f1-96a8-c6a8d0dc3241 >> Mar 16 02:35:34 node1 vdsm[26151]: WARN Attempting to remove a non existing net user: ovirtmgmt/0c3e1c08-3928-47f1-96a8-c6a8d0dc3241 >> Mar 16 02:35:34 node1 vdsm[26151]: WARN Attempting to remove a non existing network: ovirtmgmt/0c3e1c08-3928-47f1-96a8-c6a8d0dc3241 >> Mar 16 02:35:34 node1 vdsm[26151]: WARN Attempting to remove a non existing net user: ovirtmgmt/0c3e1c08-3928-47f1-96a8-c6a8d0dc3241 >> Mar 16 02:35:34 node1 vdsm[26151]: WARN File: /var/lib/libvirt/qemu/channels/0c3e1c08-3928-47f1-96a8-c6a8d0dc3241.org.qemu.guest_agent.0 already removed >> >> Please help!! > > > I'd suggest to check in /var/log/vdsm/vdsm.log why it failed to start. > > >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OTXJFCVCAHMYZ…

2 1

Error creating a storage domain (On Cisco UCS Only)
by nico.kruger＠darkmatter.ae 18 Mar '19

18 Mar '19

Hi Guys, I am trying to install a new cluster... I currently have a 9 node and two 6 node oVirt Clusters... (these were installed on 4.1 and upgraded to 4.2) So i want to build a new cluster, which is working fine on this HP notebook i use for testing (using single node gluster deployment) But when i try install this on my production servers which are Cisco UCS servers i keep getting this error: [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Error creating a storage domain]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400."} This happens during storage creation after the hosted-engine is built and after gluster has been deployed (error happens for both single and 3 replica deployments) I just cant see how an install on one type of server is successful but not on the UCS servers (which i am running my other ovirt clusters on) BTW i dont think the issue is related to Gluster Storage Create as i tried using NFS and Local storage and get the same error (on UCS server only) I am using the ovirt-node-ng-installer-4.2.0-2019011406.el7.iso install ISO Below is a tail from ovirt-hosted-engine-setup-ansible-create_storage_domain Log file 2019-01-27 11:09:49,754+0400 INFO ansible ok {'status': 'OK', 'ansible_task': u'Fetch Datacenter name', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_storage_domain.yml', 'ansible_type': 'task'} 2019-01-27 11:09:49,754+0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f9f9b7e2d50> kwargs 2019-01-27 11:09:50,478+0400 INFO ansible task start {'status': 'OK', 'ansible_task': u'Add NFS storage domain', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_storage_domain.yml', 'ansible_type': 'task'} 2019-01-27 11:09:50,479+0400 DEBUG ansible on_any args TASK: Add NFS storage domain kwargs is_conditional:False 2019-01-27 11:09:51,151+0400 DEBUG var changed: host "localhost" var "otopi_storage_domain_details_nfs" type "<type 'dict'>" value: "{ "changed": false, "skip_reason": "Conditional result was False", "skipped": true }" 2019-01-27 11:09:51,151+0400 INFO ansible skipped {'status': 'SKIPPED', 'ansible_task': u'Add NFS storage domain', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_storage_domain.yml', 'ansible_type': 'task'} 2019-01-27 11:09:51,151+0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f9f9b7e2610> kwargs 2019-01-27 11:09:51,820+0400 INFO ansible task start {'status': 'OK', 'ansible_task': u'Add glusterfs storage domain', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_storage_domain.yml', 'ansible_type': 'task'} 2019-01-27 11:09:51,821+0400 DEBUG ansible on_any args TASK: Add glusterfs storage domain kwargs is_conditional:False 2019-01-27 11:10:02,045+0400 DEBUG var changed: host "localhost" var "otopi_storage_domain_details_gluster" type "<type 'dict'>" value: "{ "changed": false, "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_ovirt_storage_domain_payload_Xous24/__main__.py\", line 682, in main\n ret = storage_domains_module.create()\n File \"/tmp/ansible_ovirt_storage_domain_payload_Xous24/ansible_ovirt_storage_domain_payload.zip/ansible/module_utils/ovirt.py\", line 587, in create\n **kwargs\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 24225, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n return future.wait() if wait else future\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n return self._code(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n self._check_fault(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n self._raise_error(response, body)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400.\n", "failed": true, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Error creating a storage domain]\". HTTP response code is 400." }" 2019-01-27 11:10:02,045+0400 DEBUG var changed: host "localhost" var "ansible_play_hosts" type "<type 'list'>" value: "[]" 2019-01-27 11:10:02,045+0400 DEBUG var changed: host "localhost" var "play_hosts" type "<type 'list'>" value: "[]" 2019-01-27 11:10:02,045+0400 DEBUG var changed: host "localhost" var "ansible_play_batch" type "<type 'list'>" value: "[]" 2019-01-27 11:10:02,046+0400 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Add glusterfs storage domain', 'ansible_result': u'type: <type \'dict\'>\nstr: {\'_ansible_parsed\': True, u\'exception\': u\'Traceback (most recent call last):\\n File "/tmp/ansible_ovirt_storage_domain_payload_Xous24/__main__.py", line 682, in main\\n ret = storage_domains_module.create()\\n File "/tmp/ansible_ovirt_storage_domain_payload_Xous24/ansible_ovirt_storage_domain_pay', 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_storage_domain.yml'} 2019-01-27 11:10:02,046+0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f9f9b859f50> kwargs ignore_errors:None 2019-01-27 11:10:02,048+0400 INFO ansible stats {'status': 'FAILED', 'ansible_playbook_duration': 39.840701, 'ansible_result': u"type: <type 'dict'>\nstr: {u'localhost': {'unreachable': 0, 'skipped': 2, 'ok': 13, 'changed': 0, 'failures': 1}}", 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/create_storage_domain.yml', 'ansible_type': 'finish'} 2019-01-27 11:10:02,048+0400 DEBUG ansible on_any args <ansible.executor.stats.AggregateStats object at 0x7f9f9da59a50> kwargs This issue is driving me crazy... any assistance on what i can do would be greatly appreciated Nico

5 8

vdsm should decouple with managed glusterfs services
by levin＠mydream.com.hk 18 Mar '19

18 Mar '19

Hi, I had experience two time of 3-node hyper-converged 4.2.8 ovirt cluster total outage due to vdsm reactivate the unresponsive node, and cause the multiple glusterfs daemon restart. As a result, all VM was paused and some of disk image was corrupted. At the very beginning, one of the ovirt node was overloaded due to high memory and CPU, the hosted-engine have trouble to collect status from vdsm and mark it as unresponsive and it start migrate the workload to healthy node. However, when it start migrate, second ovirt node being unresponsive where vdsm try reactivate the 1st unresponsive node and restart it's glusterd. So the gluster domain was acquiring the quorum and waiting for timeout. If 1st node reactivation is success and every other node can survive the timeout, it will be an idea case. Unfortunately, the second node cannot pick up the VM being migrated due to gluster I/O timeout, so second node at that moment was marked as unresponsive, and so on... vdsm is restarting the glusterd on the second node which cause disaster. All node are racing on gluster volume self-healing, and i can't mark the cluster as maintenance mode as well. What I can do is try to resume the paused VM via virsh and issue shutdown for each domain, also hard shutdown for un-resumable VM. After number of VM shutdown and wait the gluster healing completed, the cluster state back to normal, and I try to start the VM being manually stopped, most of them can be started normally, but number of VM was crashed or un-startable, instantly I found the image files of un-startable VM was owned by root(can't explain why), and can be restarted after chmod. Two of them still cannot start with "bad volume specification" error. One of them can start to boot loader, but the LVM metadata were lost. The impact was huge when vdsm restart the glusterd without human invention.

3 3

Enabled Base and Update repos?
by nico.kruger＠darkmatter.ae 18 Mar '19

18 Mar '19

Hi Guys, Should Centos Base and Update repos be enabled on ovirt hosts or only the dependancies and ovirt repos (the ones from rpm release)

1 0

Libgfapisupport messes disk image ownership
by Hesham Ahmed 18 Mar '19

18 Mar '19

I had reported this here: https://bugzilla.redhat.com/show_bug.cgi?id=1687126 Has anyone else faced this with 4.3.1?

3 3

Re: Ovirt 4.3.1 problem with HA agent
by Strahil 17 Mar '19

17 Mar '19

Hi Simone, I have noticed that my Engine's root disk is 'vda' just in standalone KVM. I have the feeling that was not the case before. Can someone check a default engine and post the output of lsblk ? Thanks in advance. Best Regards, Strahil NikolovOn Mar 15, 2019 12:46, Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: > > > On Fri, Mar 15, 2019 at 8:12 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Ok, >> >> I have managed to recover again and no issues are detected this time. >> I guess this case is quite rare and nobody has experienced that. > > > >Hi, > >can you please explain how you fixed it? > > I have set again to global maintenance, defined the HostedEngine from the old xml (taken from old vdsm log) , defined the network and powered it off. > Set the OVF update period to 5 min , but it took several hours until the OVF_STORE were updated. Once this happened I restarted the ovirt-ha-agent ovirt-ha-broker on both nodes.Then I powered off the HostedEngine and undefined it from ovirt1. > > then I set the maintenance to 'none' and the VM powered on ovirt1. > In order to test a failure, I removed the global maintenance and powered off the HostedEngine from itself (via ssh). It was brought back to the other node. > > In order to test failure of ovirt2, I set ovirt1 in local maintenance and removed it (mode 'none') and again shutdown the VM via ssh and it started again to ovirt1. > > It seems to be working, as I have later shut down the Engine several times and it managed to start without issues. > > I'm not sure this is related, but I had detected that ovirt2 was out-of-sync of the vdsm-ovirtmgmt network , but it got fixed easily via the UI. > > > > Best Regards, > Strahil Nikolov

1 0

Are people still experiencing issues with GlusterFS on 4.3x?
by Jayme 17 Mar '19

17 Mar '19

I along with others had GlusterFS issues after 4.3 upgrades, the failed to dispatch handler issue with bricks going down intermittently. After some time it seemed to have corrected itself (at least in my enviornment) and I hadn't had any brick problems in a while. I upgraded my three node HCI cluster to 4.3.1 yesterday and again I'm running in to brick issues. They will all be up running fine then all of a sudden a brick will randomly drop and I have to force start the volume to get it back up. Have any of these Gluster issues been addressed in 4.3.2 or any other releases/patches that may be available to help the problem at this time? Thanks!

7 12

Re: Broken dependencies in CentOS7 hosts upgrading to 4.2.8 from 4.2.7
by Strahil 17 Mar '19

17 Mar '19

Here is my repolist: Last login: Sun Mar 17 12:42:53 2019 from 192.168.1.43 [root@ovirt1 ~]# yum repolist Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product- : id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.wwfx.net * extras: mirror.wwfx.net * ovirt-4.3: ftp.nluug.nl * ovirt-4.3-epel: mirror.t-home.mk * updates: mirror.wwfx.net repo id repo name statusbase/7/x86_64 CentOS-7 - Base 10,019centos-sclo-rh-release/x86_64 CentOS-7 - SCLo rh 8,113extras/7/x86_64 CentOS-7 - Extras 371ovirt-4.3/7 Latest oVirt 4.3 Release 265ovirt-4.3-centos-gluster5/x86_64 CentOS-7 - Gluster 5 112ovirt-4.3-centos-opstools/x86_64 CentOS-7 - OpsTools - release 853ovirt-4.3-centos-ovirt43/x86_64 CentOS-7 - oVirt 4.3 287ovirt-4.3-centos-qemu-ev/x86_64 CentOS-7 - QEMU EV 71ovirt-4.3-epel/x86_64 Extra Packages for Enterprise Linux 7 12,981ovirt-4.3-virtio-win-latest virtio-win builds roughly matching wh 40sac-gluster-ansible/x86_64 Copr repo for gluster-ansible owned b 18updates/7/x86_64 CentOS-7 - Updates 1,163repolist: 34,293 Uploading Enabled Repositories Report Cannot upload enabled repos report, is this client registered? [root@ovirt1 ~]# As you can see my CentOS7 - Updates is enabled. So far I have managed to upgrade 4.2.7 -> 4.2.8 -> 4.3.0 -> 4.3.1 There were some issues , but nothing related to the repos. Best Regards, Strahil NikolovOn Mar 15, 2019 18:57, Roberto Nunin <robnunin(a)gmail.com> wrote: > > Hi > I have some oVirt clusters, in various config. > > One cluster based on CentOS7 hosts, another based on ovirt-node-ng. > While the second was successfully updated from 4.2.7 to 4.2.8, attempts to update hosts of the first one ends with: > > Error: Package: vdsm-4.20.46-1.el7.x86_64 (ovirt-4.2) > Requires: libvirt-daemon-kvm >= 4.5.0-10.el7_6.3 > Installed: libvirt-daemon-kvm-4.5.0-10.el7.x86_64 (@base) > libvirt-daemon-kvm = 4.5.0-10.el7 > > Being a CentOS7 installation and not an ovirt-node-ng, I cannot follow the notice in : > https://lists.ovirt.org/archives/list/users@ovirt.org/message/AJUBXAIGXVD5U… > > The only way to have libvirt-daemon-kvm release 4-5-0.10_6.4 (and not 6.3) is to enable CentOS Updates repo. Looking at host-deploy log looks fine, but It's safe to enable that repo ? There is another, safest method to update these hosts to the latest version of 4.2 ? > > Thanks in advance > > -- > Roberto Nunin > > >

2 1

GPU passthrough
by Darin Schmidt 16 Mar '19

16 Mar '19

If I'm reading through the documents correctly, you can't use consumer grade gpus such as gtx 1080's and assign them to a VM? I'm trying to do something similar to how Linus used untaid to build several gaming machines from one. Any help would be appreciated. Thanks.

2 2

single server, self hosted engine on host, one vm wont start
by abzstrak＠gmail.com 16 Mar '19

16 Mar '19

I'm not an expert with ovirt by any means, I've been running it for about 1.5 years coming from Xen. I have a Vm with 16TiB of storage that will not boot, it has some snapshots that wont remove. The web gui says it finished removing the last snapshot, but I don't think it actually did (took it about 50 hours).... I have no idea what is wrong with the vm that wont boot, it just says it's starting and never actually starts. Eventually I'll get a message that it failed to start. The boot drive of that vm is 16GB. I removed any snapshots from it and was able to clone it to a new disk, make a new vm and attach that disk to the new vm... booted fine. Now, the storage disk is what I really need working, but I do not have enough space to clone it and I cannot unattach it from the flakey vm since it has snapshots... and it seemingly won't remove the snapshots. Right now, I think its actually still maybe attempting to remove the snapshot? in top, I see qemu-img pulling all of one core at about 100%. unfortunately its very very slow because it's only using one core (of 16). This seemed to be a thing, AFAIK, when removing/merging snapshots in that it will only run single threaded... if it takes a while I don't care, I just want to know what's going on. I can't find any logs to look at that tell me that a snapshot removal or merge is ongoing or ever happened. I cannot find any logs that ever stated the progress of any such merge either. Any pointers there? Also, my backup machine died about 2 weeks ago taking its raid 5 with it... of course... I've been busy at work and figured I'd fix or rebuild it in a couple of weeks... so I don't have a good recent backup of my data on this vm, which is why I'm trying to get it to boot up again.

1 0