- Users - oVirt List Archives

Storage domain 'Inactive' but still functional
by Martijn Grendelman 18 Jul '19

18 Jul '19

Hi, For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I need some help. Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a disk image off of it, to another domain. This failed, and somewhere in the process, the whole storage domain went to status 'Inactive'. From engine.log: 2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'. 2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2019-07-17 16:30:35,410+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) was deactivated by system because it's not visible by any of the hosts. The thing is, the domain is still functional on all my hosts. It carries over 50 disks, and all involved VMs are up and running, and don't seem to have any problems. Also, 'iscsiadm' on all hosts seems to indiciate that everything is fine with this specific target and reading from the device with dd, or getting its size with 'blockdev' all works without issue. When I try to reactivate the domain, these errors are logged: 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',), code = 358 2019-07-18 09:34:53,648+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) by martijn@-authz On the SPM host, there are errors that indicate problems with the LVM volume group: 2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429', spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48) 2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool] sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78 (sp:1125) 2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=[' /dev/mapper/23536316636393463: Checksum error at offset 2748693688832', " Couldn't read volume group metadata from /dev/mapper/23536316636393463.", ' Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/23536316636393463', ' Failed to scan VG from /dev/mapper/23536316636393463', ' Volume group "875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted: "Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83) I need help getting this storage domain back online. Can anyone here help me? If you need any additional information, please let me know! Best regards, Martijn Grendelman ISAAC

3 3

assign vm traffic to a physical NIC and Storage traffic to another NIC
by Erick Perez 18 Jul '19

18 Jul '19

Hi, fresh install I have created a network called "storage" and another called "vms" and both have vlan 102 tag. Only the "vms" network have the check on "vm network". then on Compute---Hosts---hvm001---setup_host_network physical nic enp3s0f0 has ovirtmgmt and vms networks defined physical nic enp3s0f1 has storage network defined Question is how do I tell ovirt that I want the STORAGE traffic to use enp3s0f1 ? I am using NFS data domains and I cannot find a place to tell the network/physical nic my NFS traffic should use. thanks,

2 1

LDAP Group issue with rfc2307bis
by Timmi 18 Jul '19

18 Jul '19

Hi oVirt List, I'm currently working on my new oVirt setup and want to integrate it into our LDAP server. Accounts are working fine but I have problems to get the groups working correctly. The LDAP server is base on ClearOS which is using the rfc2307bis setup. Means I don't have MemberOf inside my users. The user DN is as Member inside the group. I manage that oVirt is able to read the groups while overwriting: search.rfc2307-resolve-groups-memberUid.search-request.filter = &(objectClass=posixGroup)(memberUid=${seq:_rfc2307_uid_encoded}) with search.rfc2307-resolve-groups-memberUid.search-request.filter = &(objectClass=posixGroup)(member=${seq:_rfc2307_dn}) This is working absolutely fine for my admin group in "Administrator Portal". I can asign the group to the system permission "SuperUser" and everything is working great. My problem is with the "VM Portal" I have assigned "PowerUser" rights to a quota and it is possible to login but I receive the following error in the engine.log. 2019-07-18 07:38:12,317+02 ERROR [org.ovirt.engine.core.bll.GetPermissionsForObjectQuery] (default task-5) [a6828f8b-8ded-422f-a216-5e5406d7bf20] Query execution failed due to insufficient permissions. 2019-07-18 07:38:12,319+02 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-5) [] Operation Failed: query execution failed due to insufficient permissions. I'm able to see the group permission in the user details. So I guess that something is already working. But I guess the error is preventing me to have the "create VM" button on the "VM Portal". Would be great if someone could help me out. I'm running the latest 4.3.4 version. Best regards Christoph

1 0

Support and suggestions for 10/25 GbE cards
by Gianluca Merlo 17 Jul '19

17 Jul '19

Hello everyone, I am in the hardware selection phase for an oVirt cluster which has a requirement for 10/25 GbE connectivity. Does anyone have any advice on which adapters are better supported and working on oVirt Node? The choice I have is between: - Intel XXV710 - Mellanox ConnectX-4 It is my first foray into this tier of connectivity, so pardon the naivety of the question. I would just like to hear suggesions from the experience any of you may have had in the field, so I can be better prepared and know which kind of problems I may run into, and if choosing a specific card over the other can spare me some. Thank you in advance for your help Gianluca

1 0

hosted engine, Hyperconverged recovery
by William Kwan 17 Jul '19

17 Jul '19

Hi, I need some direction to make sure we won't make more mistake in recovering a 3-node self hosted engine with Gluster. Someone carelessly, very, wiped out the first few MB of the LVM PV on one node. We changed the quorum and brought up the Gluster and oVirt. We ran vgcfgrestore on the 'bad node' and we can run pvscan, vgscan and lvscan to see the LV's. What should I do next to prevent more damage or corrupt the glusterfs on the other two nodes? Will this work? mkfs the LV's bring up gluster run gluster sync <goodNode>? ThanksWill

3 2

VM says paused, but does not appear to be.
by Jacob Green 17 Jul '19

17 Jul '19

I have a Fibre Channel back end for my oVirt cluster, and one of my VMS is saying "paused due to lack of storage space", however I can confirm that the back end storage has plenty of space and the VM itself does not appear to be paused. I can SSH the VM and perform tasks on the VM. So I am trying to figure out why it would say that, and how to clear it. Also have 20 something other VMS on this cluster and no errors like this on those. I am running engine 4.3.3.7-1.el7 Also I do not fully understand this chart, what I can tell you is that the top storage domain is my Fibre Channel storage, and it has plenty of available space, 35%-40% free And lastly this is what a df -h looks like on the given VM. If anyone has some insight on to what might cause this let me know and ill chase it down. Thank you

1 0

Re: Hardware Used
by Strahil 17 Jul '19

17 Jul '19

As far as I know several orgs are using AMD Ryzen CPUs with oVirt. Ryzen CPUs are quite nice from price/performance perspective. Best Regards, Strahil NikolovOn Jul 17, 2019 15:04, Michal Skrivanek <michal.skrivanek(a)redhat.com> wrote: > > > >> On 16 Jul 2019, at 15:10, Adrian Odendaal <adrian(a)quodes.co.za> wrote: >> > No well we saw that only CPU’s are supported. > > So we want to find out what hardware we need in order to progress. > > Servers with the needed CPU architecture. > > Not sure what you’re looking for. A developer kind of answer - you got that already. If you want anything better I suggest you talk to a vendor who sells oVirt for a living and help you with the right architecture fitting your needs. > List of certified Intel CPUs for RHEL (and hence CentOS) is here [1] and here[2]. We also support IBM POWER8 and POWER9, and if you wanna start something really big, IBM zSystem mainframes as a tech preview;-) > > Thanks, > michal > > [1] https://access.redhat.com/support/policy/intel > [2] https://access.redhat.com/support/policy/amd > >> >> Kind Regards >> >> >> <bcard.jpg> >> >> >> >> >> >> Legal Disclaimer:

2 1

Re: Hardware Used
by Emil Natan 17 Jul '19

17 Jul '19

I think the right place for this question is the ovirt users mailing list. Added. On Mon, Jul 15, 2019 at 10:28 AM <adrian(a)quodes.co.za> wrote: > Hi All > > We are looking into setting up a highly scalable and HA Ovirt > infrastructure but I see only some CPU's are supported. Is this only for th > Hypervisors or for the entire cluster? > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/X5MMJXN67LMBC… > -- Emil Natan RHV/CNV DevOps

3 6

Re: UI exception when changing Cluster in New VM window
by Strahil 17 Jul '19

17 Jul '19

Dear Sharon, I think you mean Marko Vrgotic. Best Regards, Strahil NikolovOn Jul 10, 2019 14:19, Sharon Gratch <sgratch(a)redhat.com> wrote: > > Hi Strahil, > > Based on the stuck trace in the UI log that you sent, I am pretty sure that this UI exception was already fixed on https://gerrit.ovirt.org/#/c/99463/ for oVirt 4.3.4. > So please follow Michal's suggestion to install latest 4.3.5 version for overcoming this issue. > > Thanks, > Sharon > > On Tue, Jul 9, 2019 at 8:19 PM Michal Skrivanek <michal.skrivanek(a)redhat.com> wrote: >> >> can you please try that on 4.3.5 and if it persist open a bug with that ui.log attached? >> >> thanks, >> michal >> >>> On 5 Jul 2019, at 16:06, Vrgotic, Marko <M.Vrgotic(a)activevideo.com> wrote: >>> >>> Hi Strahil, >>> >>> I tried what you said and I am still getting an exception. >>> >>> Correct me if I am wrong, but my expectation is following, when creating new VM: >>> Select cluster >>> Whether I want to use template or create brand new VM from scratch, list of templates and everything else specific to cluster I selected should be loaded >>> >>> Instead, I get the exception and have to refresh/reload the page. >>> >>> In short, when creating new VM, I am currently locked to a default loaded cluster, which in my case is avshared1. >>> >>> To give you bit more insight into configuration, here are the cluster details: >>> >>> <image001.png> >>> >>> Per cluster:

2 1

Re: ovirt_disk and ubuntu issues
by Vrgotic, Marko 16 Jul '19

16 Jul '19

Dear oVIrt, Even though I would like to get some insight into what could be reason this is no working, I did find a workaround: Instead of trying to get Ubuntu disk specified with ovirt_disk size, I used qemu-img resize to increase the disk size before importing it to oVIrt. This works, but it still going to present the problem if User eventually wants to increase for example disk from 40GBto 80GB. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic Sr. System Engineer @ System Administration m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> Date: Wednesday, 10 July 2019 at 16:19 To: "users(a)ovirt.org" <users(a)ovirt.org> Subject: ovirt_disk and ubuntu issues Dear oVirt, I am downloading the ubuntu cloud image 16.04 and or 18.04: - name: "Download base cloud image from server" get_url: url: "{{ image_url }}" checksum: "sha256:{{ image_checksum }}" validate_certs: yes dest: "/tmp/{{ inventory_hostname_short }}.qcow2" delegate_to: localhost creating a 40GB HDD and attaching image to it: - name: "Create oVirt disk with base image (with 40Gb allocated)" ovirt_disk: name: "{{ inventory_hostname_short }}" interface: virtio size: 40GiB format: cow upload_image_path: "/tmp/{{ inventory_hostname_short }}.qcow2" storage_domain: ovirt_production wait: true delegate_to: localhost creating VM afterwards: - name: "Create new Ubuntu VMs from cloud image" delegate_to: localhost ovirt_vm: auth: "{{ ovirt_auth }}" name: "{{ inventory_hostname_short }}" disks: - name: "{{ inventory_hostname_short }}" graphical_console: protocol: vnc serial_console: true usb_support: true soundcard_enabled: false operating_system: "{{ operating_system_type }}" type: server nics: - name: nic1 profile_name: tenant1 interface: virtio nic_on_boot: true cloud_init: host_name: "{{ inventory_hostname }}" user_name: ubuntu authorized_ssh_keys: "{{ ssh_agent_pubkeys.stdout }}" state: "running" cluster: "{{ ovirt_cluster }}" when: inventory_hostname in groups['ubuntu-baker'] When VM gets created, I can see in oVIrt VM details disk created is 40GB. Executing df -h, gives me following: root@av3-ubuntu-18-base:/home/ubuntu# df -h Filesystem Size Used Avail Use% Mounted on udev 447M 0 447M 0% /dev tmpfs 92M 696K 92M 1% /run /dev/vda1 2.0G 1.3G 706M 65% / tmpfs 460M 0 460M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 460M 0 460M 0% /sys/fs/cgroup /dev/vda15 105M 3.6M 101M 4% /boot/efi tmpfs 92M 0 92M 0% /run/user/1000 Initially I thought growpart or resize2fs is not triggered, but then running dmesg or fdisk /dev/vda, told me that physical disk size is still only size of the downloaded ubuntu cloud image. Disk /dev/vda: 2.2 GiB, 2361393152 bytes, 4612096 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Is this related to ovirt_disk module, or ubuntu and ovirt_disk, since I do not have this behavior with CentOS 7 images? Can you advise how to proceed, in case I am missing some configuration parameter or command to be run? The following Ubuntu images are used: ubuntu-16: image_url=https://cloud-images.ubuntu.com/xenial/current/xenial-server-clou… image_checksum=fda868058586b129c7fdb6472fe575e911f7c67551a6dc75966f2ec02201bdae ubuntu-18: image_url=https://cloud-images.ubuntu.com/bionic/current/bionic-server-clou… image_checksum=7d2b90022a169119d7726c0fefa1713acbead7cc36d282c879896fd89c5a6663 Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic Sr. System Engineer @ System Administration m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> tel. +31 (0)35 677 4131

2 4