November 2018 - Users - oVirt List Archives

Ovirt Maximums
by Brian Wilson 13 Dec '18

13 Dec '18

Is there a resource available that would describe the configurable maximums for the Ovirt Engine and Ovirt Nodes. Things like: How many Logical Networks per host? If one does not exist does anyone have any experiences with reaching limits that caused issues specifically with regards to the number of networks on hosts.

2 2

VM ramdomly unresponsive
by fsoyer 08 Dec '18

08 Dec '18

Hi all, I continue to try to understand my problem between (I suppose) oVirt anf Gluster. After my recents posts titled 'VMs unexpectidly restarted' that did not provide solution nor search idea, I submit to you another (related ?) problem. Parallely with the problem of VMs down (that did not reproduce since Oct 16), I have ramdomly some events in the GUI saying "VM xxxxx is not responding." For example, VM "patjoub1" on 2018-11-11 14:34. Never the same hour, not all the days, often this VM patjoub1 but not always : I had it on two others. All VMs disks are on a volume DATA02 (with leases on the same volume). Searching in engine.log, I found : 2018-11-11 14:34:32,953+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [] VM '6116fb07-096b-4c7e-97fe-01ecc9a6bd9b'(patjoub1) moved from 'Up' --> 'NotResponding' 2018-11-11 14:34:33,116+01 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [] Invalid or unknown guest architecture type '' received from guest agent 2018-11-11 14:34:33,176+01 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [] EVENT_ID: VM_NOT_RESPONDING(126), VM patjoub1 is not responding. ... ... 2018-11-11 14:34:48,278+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [] VM '6116fb07-096b-4c7e-97fe-01ecc9a6bd9b'(patjoub1) moved from 'NotResponding' --> 'Up'So it becomes up 15s after, and the VM (and the monitoring) see no downtime. At this time, I see in vdsm.log of the nodes : 2018-11-11 14:33:49,450+0100 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_DATA02/ffc53fd8-c5d1-4070-ae51-2e91835cd937/dom_md/metadata (monitor:498) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 496, in _pathChecked delay = result.delay() File "/usr/lib/python2.7/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) MiscFileReadException: Internal file read failure: (u'/rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_DATA02/ffc53fd8-c5d1-4070-ae51-2e91835cd937/dom_md/metadata', 1, 'Read timeout') 2018-11-11 14:33:49,450+0100 INFO (check/loop) [storage.Monitor] Domain ffc53fd8-c5d1-4070-ae51-2e91835cd937 became INVALID (monitor:469) 2018-11-11 14:33:59,451+0100 WARN (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_DATA02/ffc53fd8-c5d1-4070-ae51-2e91835cd937/dom_md/metadata' is blocked for 20.00 seconds (check:282) 2018-11-11 14:34:09,480+0100 INFO (event/37) [storage.StoragePool] Linking /rhev/data-center/mnt/glusterSD/victor.local.systea.fr:_DATA02/ffc53fd8-c5d1-4070-ae51-2e91835cd937 to /rhev/data-center/6efda7f8-b62f-11e8-9d16-00163e263d21/ffc53fd8-c5d1-4070-ae51-2e91835cd937 (sp:1230)OK : so, DATA02 marked as blocked for 20s ? I definitly have a problem with gluster ? I'll inevitably find the reason in the gluster logs ? Uh : not at all. Please see gluster logs here : https://seafile.systea.fr/d/65df86cca9d34061a1e4/ Unfortunatly I discovered this morning that I have not the sanlock.log for this date. I don't understand why, the log rotate seems OK with "rotate 3", but I have no backups files :(. But, luck in bad luck, the same event occurs this morning ! Same VM patjoub1, 2018-11-13 08:01:37. So I have added the sanlock.log for today, maybe it can help. IMPORTANT NOTE : don't forget that Gluster log with on hour shift. For this event at 14:34, search at 13h34 in gluster logs. I recall my configuration : Gluster 3.12.13 oVirt 4.2.3 3 nodes where the third is arbiter (volumes in replica 2) The nodes are never overloaded (CPU average 5%, no peak detected at the time of the event, mem 128G used at 15% (only 10 VMs on this cluster)). Network underused, gluster is on a separate network on a bond (2 NICs) 1+1Gb mode 4 = 2Gb, used in peak at 10%. Here is the configuration for the given volume : # gluster volume status DATA02 Status of volume: DATA02 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick victorstorage.local.systea.fr:/home/d ata02/data02/brick 49158 0 Y 4990 Brick gingerstorage.local.systea.fr:/home/d ata02/data02/brick 49153 0 Y 8460 Brick eskarinastorage.local.systea.fr:/home /data01/data02/brick 49158 0 Y 2470 Self-heal Daemon on localhost N/A N/A Y 8771 Self-heal Daemon on eskarinastorage.local.s ystea.fr N/A N/A Y 11745 Self-heal Daemon on victorstorage.local.sys tea.fr N/A N/A Y 17055 Task Status of Volume DATA02 ------------------------------------------------------------------------------ There are no active volume tasks # gluster volume info DATA02 Volume Name: DATA02 Type: Replicate Volume ID: 48bf5871-339b-4f39-bea5-9b5848809c83 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: victorstorage.local.systea.fr:/home/data02/data02/brick Brick2: gingerstorage.local.systea.fr:/home/data02/data02/brick Brick3: eskarinastorage.local.systea.fr:/home/data01/data02/brick (arbiter) Options Reconfigured: network.ping-timeout: 30 server.allow-insecure: on cluster.granular-entry-heal: enable features.shard-block-size: 64MB performance.stat-prefetch: on server.event-threads: 3 client.event-threads: 8 performance.io-thread-count: 32 storage.owner-gid: 36 storage.owner-uid: 36 user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: enable performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.server-quorum-ratio: 51% So : is there someone around trying to make me understand what append ? Pleeease :/ -- Regards, Frank

3 4

Cloud-init reset network configuration to default dhcp after reboot and regular run
by Mike Lykov 06 Dec '18

06 Dec '18

Hi All! I'm trying to configure network in VMs created from template, and see strange behaviour from cloud-init. cloud-init installed and enabled in template. ver cloud-init-0.7.9-24.el7.centos.1.x86_64 guest Centos 7.5 ovirt 4.2.7 from ovirt-releases-42-pre repo 1. I use "run once" with initial run - use cloud-init - networks in-guest net iface : eth0 add new - static enter address, mask, gw ipv6 none Run (once) and cloud-init configure ifcfg-eth0 for that address (successfully). 2. I shutdown that VM and use "Run" (regular) without "use cloud init" in VM properties, awaiting that above configurations are saved (and booted with it). But because "use cloud init" not checked, and cloud-init service enabled, it start, cannot find datasource and drop configuration to default (dhcp). In cloud-init.log 2018-11-20 13:53:24,153 - util.py[WARNING]: No instance datasource found! Likely bad things to come! 2018-11-20 13:53:24,153 - util.py[DEBUG]: No instance datasource found! Likely bad things to come! Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cloudinit/cmd/main.py", line 236, in main_init init.fetch(existing=existing) File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 343, in fetch return self._get_data_source(existing=existing) File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 253, in _get_data_source pkg_list, self.reporter) File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 320, in find_source raise DataSourceNotFoundException(msg) DataSourceNotFoundException: Did not find any data source, searched classes: (DataSourceNoCloudNet) 2018-11-20 13:53:24,194 - stages.py[DEBUG]: applying net config names for {'version': 1, 'config': [{'subnets': [{'type': 'dhcp'}], 'type': 'physical', 'name': 'eth0', 'mac_address': '56:6f:21:4a:00:04'}]} It reverts config as in here https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NE27UO4WNZIC27… 3. then I found bug https://bugzilla.redhat.com/show_bug.cgi?id=1439373#c5 and when "run once" I disable network config as in that comment. Shutdown, Run (not once) and voila! Ip address are static! 2018-11-20 15:07:31,601 - handlers.py[DEBUG]: finish: init-network/search-NoCloudNet: SUCCESS: no network data found from DataSource NoCloudNet ..... 2018-11-20 15:07:31,602 - util.py[WARNING]: No instance datasource found! Likely bad things to come! 2018-11-20 15:07:31,602 - util.py[DEBUG]: No instance datasource found! Likely bad things to come! 2018-11-20 15:07:31,639 - stages.py[DEBUG]: network config disabled by system_cfg 2018-11-20 15:07:31,639 - stages.py[INFO]: network config is disabled by system_cfg 2018-11-20 15:07:31,639 - main.py[DEBUG]: [net] Exiting without datasource in local mode 2018-11-20 15:07:31,640 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False) 2018-11-20 15:07:31,640 - util.py[DEBUG]: Read 12 bytes from /proc/uptime 2018-11-20 15:07:31,640 - util.py[DEBUG]: cloud-init mode 'init' took 0.287 seconds (0.29) 2018-11-20 15:07:31,640 - handlers.py[DEBUG]: finish: init-network: SUCCESS: searching for network datasources "cloud-init used to use a "marker" file that it created on initial execution. If that "marker" file existed it would not rerun on reboot. " - are it not working in ovirt/this cloud-init version ? -- Mike

2 8

Repository ovirt-4.2 is listed more than once in the configuration
by Markus Frei 06 Dec '18

06 Dec '18

Hello everybody I am frequently receiving the following messages from ovirt: Repository ovirt-4.2-epel is listed more than once in the configuration Repository ovirt-4.2-centos-gluster312 is listed more than once in the configuration Repository ovirt-4.2-virtio-win-latest is listed more than once in the configuration Repository ovirt-4.2-centos-qemu-ev is listed more than once in the configuration Repository ovirt-4.2-centos-opstools is listed more than once in the configuration Repository centos-sclo-rh-release is listed more than once in the configuration Repository ovirt-4.2-centos-ovirt42 is listed more than once in the configuration Repository ovirt-4.2 is listed more than once in the configuration I`ve got following repos configured in directory /etc/yum.repos.d: ovirt-4.2-dependencies.repo ovirt-4.2-pre-dependencies.repo ovirt-4.2-pre.repo ovirt-4.2.repo I deleted the 'pre' repos several times but apparently they are being recreated somehow. Can anybody explain this behaviour please and eventually the meaning of these 'pre' repos in generell as well? Thank you in advance. Kind regards, Chris

2 1

Host's network configuration differs from DC
by siddhant＠trustingsocial.com 05 Dec '18

05 Dec '18

We are unable to setup host network because of some out-of-sync error.

2 1

Best Openstack version to integrate with oVirt 4.2.7
by Gianluca Cecchi 04 Dec '18

04 Dec '18

Hello, do you think it is ok to use Rocky version of Openstack to integrate its services with oVirt 4.2.7 on CentOS 7? I see on https://repos.fedorapeople.org/repos/openstack/ that, if Rocky is too new, between the older releases available there are, from newer to older: Queens Pike Ocata Newton At the moment I have two separate lab environments: oVirt with 4.2.7 Openstack with Rocky (single host with packstack allinone) just trying first integration steps with these versions, it seems I'm not able to communicate with glance, because I get in engine.log 2018-11-10 17:32:58,386+01 ERROR [org.ovirt.engine.core.bll.provider.storage.AbstractOpenStackStorageProviderProxy] (default task-51) [e2fccee7-1bb2-400f-b8d3-b87b679117d1] Not Found (OpenStack response error code: 404) Nothing in glance logs on openstack, apparently. In my test I'm using http://xxx.xxx.xxx.xxx:9292 as provider url checked the authentication check box and glance user with its password 35357 as the port and services as the tenant a telnet on port 9292 of openstack server from engine to openstack is ok similar with cinder I get: 2018-11-10 17:45:42,226+01 ERROR [org.ovirt.engine.core.bll.provider.storage.AbstractOpenStackStorageProviderProxy] (default task-50) [32a31aa7-fe3f-460c-a8b9-cc9b277deab7] Not Found (OpenStack response error code: 404) So before digging more I would lile to be certain which one is currently the best combination, possibly keeping as fixed the oVirt version to 4.2.7. Thanks, Gianluca

6 9

FCoE wont initialize on reboot oVirt 4.2.
by Jacob Green 04 Dec '18

04 Dec '18

I have HP BL460c Gen9 blades with BCM57840 NetXtreme II 10/20-Gigabit Ethernet via a HP Virtual connection. In my ovirt 4.1 environment I have fibre channel working great. However in the new environment that I want to bring the data domain over too ultimately, I am having issues with ovirt interfering with the hosts ability to see the Fiber Channel storage. If i build a clean CentOS 7 installation and get my FCoE module installed and Fiber channel set up on my appropriate interfaces, it works and sees the fiber channel interface every time I type fcoeadm -i, I can reboot a million times, does not go away. However once I turn it into a oVirt 4.2 node and add it to my environment and reboot the blade it is hit or miss if fcoeadm -i is going to return interface information. If I then type "systemctl restart network" my fiber channel comes online, but I should not need to do this. I can see in my dmesg logs that the fiber channel is initializing on boot. [ 39.465578] cnic: QLogic cnicDriver v2.5.22 (July 20, 2015) [ 39.465594] bnx2x 0000:06:00.2 eno51: Added CNIC device [ 39.475618] bnx2x 0000:06:00.3 eno52: Added CNIC device [ 39.495575] bnx2fc: QLogic FCoE Driver bnx2fc v2.11.8 (October 15, 2015) [ 39.505971] bnx2fc: bnx2fc: FCoE initialized for eno52. [ 39.506299] bnx2fc: [06]: FCOE_INIT passed [ 39.516308] bnx2fc: bnx2fc: FCoE initialized for eno51. [ 39.516654] bnx2fc: [06]: FCOE_INIT passed Reminder I have not added a FC storage domain yet, because I need to turn off and detach the domain from the old 4.1 environment first. However that should not keep the fiber channel interfaces from coming up... And I need to know its working before I do that. Below is what an fcoeadm -i should return when it seems to be working. ____________________________________________________________________________________________________ fcoeadm -i Description: BCM57840 NetXtreme II 10/20-Gigabit Ethernet Revision: 11 Manufacturer: Broadcom Limited Serial Number: 5CB901C7EE00 Driver: bnx2x 1.712.30-0 Number of Ports: 1 Symbolic Name: bnx2fc (QLogic BCM57840) v2.11.8 over eno51 OS Device Name: host10 Node Name: 0x50060b0000c27a05 Port Name: 0x50060b0000c27a04 Fabric Name: 0x1000c4f57c218ff4 Speed: unknown Supported Speed: 1 Gbit, 10 Gbit MaxFrameSize: 2048 bytes FC-ID (Port ID): 0x0a025c State: Online Description: BCM57840 NetXtreme II 10/20-Gigabit Ethernet Revision: 11 Manufacturer: Broadcom Limited Serial Number: 5CB901C7EE00 Driver: bnx2x 1.712.30-0 Number of Ports: 1 Symbolic Name: bnx2fc (QLogic BCM57840) v2.11.8 over eno52 OS Device Name: host11 Node Name: 0x50060b0000c27a07 Port Name: 0x50060b0000c27a06 Fabric Name: 0x1000c4f57c21979d Speed: unknown Supported Speed: 1 Gbit, 10 Gbit MaxFrameSize: 2048 bytes FC-ID (Port ID): 0x14037b State: Online ____________________________________________________________________________________________________ However if I reboot the node from the ovirt console and wait a few minutes after it has rebooted then type fcoeadm -i I get the following. fcoeadm -i fcoeadm: No action was taken Try 'fcoeadm --help' for more information. it is not until I perform a systemctl restart network, that I get the correct output from above. Any help or insight into fiber channel with ovirt 4.2 would be greatly appreciated. -- Jacob Green Systems Admin American Alloy Steel 713-300-5690

2 7

SPICE QXL Crashes Linux Guests
by Alex McWhirter 03 Dec '18

03 Dec '18

I'm having an odd issue that i find hard to believe could be a bug, and not some kind of user error, but im at a loss for where else to look. when booting a linux ISO with QXL SPICE graphics, the boot hangs as soon as kernel modesetting kicks in. Tried with latest debian, fedora, and centos. Sometimes it randomly works, but most often it does not. QXL / VGA VNC work fine. However if i wait a few minutes after starting the VM for the graphics to start, then there are no issues and i can install as usual. So after install, i reboot, hangs on reboot right after graphics switch back to text mode with QXL SPICE, not with VNC. So i force power off, reboot, and wait a while for it to boot. If i did text only install, when i open a spice console it will hang after typing a few characters. If i did a graphical install then as long as i waited long enough for X to start, then it works perfectly fine. I tried to capture some logs, but since the whole guest OS hangs it's rather hard to pull off. I did see an occasional error about the mouse driver, so that's really all i have to go on. As for the spice client, im using virt-viewer on windows 10 x64, tried various versions of virt-viewer just to be sure, no change. I also have a large amount on windows guests with QXL SPICE. These all work with no issue. Having guest agent installed in the linux guest seems to make no difference. There are no out of the ordinary logs on the VDSM hosts, but i can provide anything you may need. It's not specific to any one host, i have 10 VM hosts in the cluster, they all do. They are westmere boxes if that makes a difference. Any ideas on how i should approach this? VNC works well enough for text only linux guest, but not being able to reboot my GUI linux guests without also closing my spice connection is a small pain. as far as ovirt versions im on the latest, this is a rather fresh install. just set it up a few days ago, but i've been a long time ovirt user. I am using a squid spice proxy if that makes a difference.

2 2

Out of sync, hosts network config differs from DC
by femi adegoke 02 Dec '18

02 Dec '18

Property: default route, host - true, DC - false I have 4 nics. bond0 = 2 x 10g eno1 = ovirtmgmt eno2 = for vm traffic. eno2 says it's out of sync, hosts network config differs from DC, default route: host - true, DC - false. I have tried "sync all networks" but the message still remains See attached. Where can I look to fix the issue?

3 2

UPS with USB Support
by florentl 01 Dec '18

01 Dec '18

Hi everybody, I'm currently setting up an ovirt solution. I have three servers with glusterfs. They run hosted engine. I configured the power management of the node for using idrac agent (I have Dell servers). The communication is ok but I don't really know if configuring power management is very usefull with only three servers. My three servers are powered via an ups which have only one usb port. I'm wondering what is the best option to shutdown the datacenter in case of power failure. I have read in previous posts that I have to write a script. So I think I'm going to : - install apcupsd on the node connected with the ups via usb - configure apcupsd.conf to lanch a script and send a mail - write a script to shutdown properly all the wm and to put the datacenter in maintenance mode What do you think about this solution ? Is there a better way to manage the vm's and datacenter's shutdown with the power management solution associated with ovirt host ? Regards, Florent

3 2