January 2019 - Users - oVirt List Archives

Ovirt cluster unstable; gluster to blame (again)
by Jim Kusznir 04 Feb '19

04 Feb '19

hi all: Once again my production ovirt cluster is collapsing in on itself. My servers are intermittently unavailable or degrading, customers are noticing and calling in. This seems to be yet another gluster failure that I haven't been able to pin down. I posted about this a while ago, but didn't get anywhere (no replies that I found). The problem started out as a glusterfsd process consuming large amounts of ram (up to the point where ram and swap were exhausted and the kernel OOM killer killed off the glusterfsd process). For reasons not clear to me at this time, that resulted in any VMs running on that host and that gluster volume to be paused with I/O error (the glusterfs process is usually unharmed; why it didn't continue I/O with other servers is confusing to me). I have 3 servers and a total of 4 gluster volumes (engine, iso, data, and data-hdd). The first 3 are replica 2+arb; the 4th (data-hdd) is replica 3. The first 3 are backed by an LVM partition (some thin provisioned) on an SSD; the 4th is on a seagate hybrid disk (hdd + some internal flash for acceleration). data-hdd is the only thing on the disk. Servers are Dell R610 with the PERC/6i raid card, with the disks individually passed through to the OS (no raid enabled). The above RAM usage issue came from the data-hdd volume. Yesterday, I cought one of the glusterfsd high ram usage before the OOM-Killer had to run. I was able to migrate the VMs off the machine and for good measure, reboot the entire machine (after taking this opportunity to run the software updates that ovirt said were pending). Upon booting back up, the necessary volume healing began. However, this time, the healing caused all three servers to go to very, very high load averages (I saw just under 200 on one server; typically they've been 40-70) with top reporting IO Wait at 7-20%. Network for this volume is a dedicated gig network. According to bwm-ng, initially the network bandwidth would hit 50MB/s (yes, bytes), but tailed off to mostly in the kB/s for a while. All machines' load averages were still 40+ and gluster volume heal data-hdd info reported 5 items needing healing. Server's were intermittently experiencing IO issues, even on the 3 gluster volumes that appeared largely unaffected. Even the OS activities on the hosts itself (logging in, running commands) would often be very delayed. The ovirt engine was seemingly randomly throwing engine down / engine up / engine failed notifications. Responsiveness on ANY VM was horrific most of the time, with random VMs being inaccessible. I let the gluster heal run overnight. By morning, there were still 5 items needing healing, all three servers were still experiencing high load, and servers were still largely unstable. I've noticed that all of my ovirt outages (and I've had a lot, way more than is acceptable for a production cluster) have come from gluster. I still have 3 VMs who's hard disk images have become corrupted by my last gluster crash that I haven't had time to repair / rebuild yet (I believe this crash was caused by the OOM issue previously mentioned, but I didn't know it at the time). Is gluster really ready for production yet? It seems so unstable to me.... I'm looking at replacing gluster with a dedicated NFS server likely FreeNAS. Any suggestions? What is the "right" way to do production storage on this (3 node cluster)? Can I get this gluster volume stable enough to get my VMs to run reliably again until I can deploy another storage solution? --Jim

11 23

ETL service aggregation to hourly tables has encountered an error. Please consult the service log for more details.
by melnyksergii＠gmail.com 04 Feb '19

04 Feb '19

Dears, I have an a some error in Ovirt 4.2.7 In dash I see: ETL service aggregation to hourly tables has encountered an error. Please consult the service log for more details. In log ovirt engine server: 2019-01-14 15:59:59|rwL6AB|euUXph|wfcjQ7|OVIRT_ENGINE_DWH|HourlyTimeKeepingJob|Default|5|tWarn|tWarn_1|2019-01-14 15:59:59| ETL service aggregation to hourly tables has encountered an error. lastHourAgg value =Mon Jan 14 14:00:00 EET 2019 and runTime = Mon Jan 14 15:59:59 EET 2019 .Please consult the service log for more details.|42 In some sources people said the problem is in PostgreSQL DB, but I don't understand how can I fix this problem? Thanks

4 9

oVirt can't communicate with vSphere
by Benjamin Selinger 03 Feb '19

03 Feb '19

Trying to get my ovirt cluster connected to my vSphere cluster to import my guests. When trying to connect, I get: "VDSM ovirt1 command GetVmsNamesFromExternalProviderVDS failed: internal error: curl_easy_perform() returned an error: Couldn't connect to server (7) : Failed connect to 10.0.0.55:443; Connection timed out" ovirt1 node (192.168.1.195) is at location A while vSphere (10.0.0.55) is at location B. I added a static route on ovirt1 for 10.0.0.0/8 via 192.168.1.13, which has a VPN connection back to 10.0.0.0/8. ICMP from ovirt1 gets through just fine, but any other traffic never leaves ovirt1 (ie: 443 traffic never arrives at 1.13). I'm assuming there's some firewall rule somewhere, blocking anything other than outbound ICMP, but I have been unable to find it. Any suggestions?

2 3

Mounting ISO in subfolder
by Giulio Casella 01 Feb '19

01 Feb '19

Hi, I noticed that if I create a directory in root of ISO domain and put an image in it, you can see it in admin portal image list (listed as "foo/bar.iso"), but mounting that iso in a VM ("change CD") fails "Error while executing action Change CD: Drive image file could not be found" I can easily reproduce this behaviour in ovirt 4.2.8 and rhv 4.2.7. Is it a bug? Cheers, gc

5 9

Storage Domain Ovirt 4.3 RC EPYC
by Darin Schmidt 01 Feb '19

01 Feb '19

Uncaught exception occurred. Please try reloading the page. Details: (TypeError) : Cannot read property 'Vg' of null Please have your administrator check the UI logs Im trying to setup Ovirt 4.3 RC because its the only one that supports AMD EPYC. The issue Im having is that I creates a raid 5 array of 5 120GB SSDs using mdadm and mounted it at /VM. I put the Host into maintenance, select Configure Local Storage, and it fails telling me: New Local Storage Domain: Storage format is unsupported I formatted the array to ext4. This is a self hosted system. Anyone have any ideas? [root@ovirt VM]# uname -r 3.10.0-957.1.3.el7.x86_64 [root@ovirt VM]# cat /etc/centos-release CentOS Linux release 7.6.1810 (Core) [root@ovirt VM]# rpm -qa|grep ovirt ovirt-engine-backend-4.3.0.2-1.el7.noarch ovirt-imageio-common-1.4.6-1.el7.x86_64 ovirt-engine-setup-plugin-websocket-proxy-4.3.0.4-1.el7.noarch ovirt-ansible-vm-infra-1.1.12-1.el7.noarch ovirt-ansible-infra-1.1.11-1.el7.noarch ovirt-engine-setup-plugin-ovirt-engine-common-4.3.0.4-1.el7.noarch libgovirt-0.3.4-1.el7.x86_64 ovirt-hosted-engine-setup-2.3.3-1.el7.noarch ovirt-iso-uploader-4.3.0-1.el7.noarch ovirt-engine-4.3.0.2-1.el7.noarch python2-ovirt-host-deploy-1.8.0-1.el7.noarch ovirt-ansible-repositories-1.1.4-1.el7.noarch python-ovirt-engine-sdk4-4.3.0-2.el7.x86_64 ovirt-engine-webadmin-portal-4.3.0.2-1.el7.noarch ovirt-engine-vmconsole-proxy-helper-4.3.0.4-1.el7.noarch ovirt-vmconsole-proxy-1.0.6-2.el7.noarch ovirt-ansible-image-template-1.1.9-1.el7.noarch ovirt-ansible-shutdown-env-1.0.3-1.el7.noarch ovirt-engine-dwh-4.3.0-1.el7.noarch ovirt-ansible-roles-1.1.6-1.el7.noarch ovirt-host-deploy-common-1.8.0-1.el7.noarch ovirt-engine-api-explorer-0.0.3-1.el7.noarch ovirt-engine-setup-base-4.3.0.4-1.el7.noarch ovirt-engine-extensions-api-impl-4.3.0.4-1.el7.noarch cockpit-ovirt-dashboard-0.12.1-1.el7.noarch ovirt-engine-tools-backup-4.3.0.2-1.el7.noarch ovirt-engine-restapi-4.3.0.2-1.el7.noarch ovirt-ansible-hosted-engine-setup-1.0.8-1.el7.noarch ovirt-engine-websocket-proxy-4.3.0.4-1.el7.noarch ovirt-hosted-engine-ha-2.3.1-1.el7.noarch ovirt-ansible-cluster-upgrade-1.1.11-1.el7.noarch ovirt-cockpit-sso-0.1.1-1.el7.noarch ovirt-engine-ui-extensions-1.0.2-1.el7.noarch ovirt-engine-setup-4.3.0.4-1.el7.noarch ovirt-imageio-daemon-1.4.6-1.el7.noarch ovirt-host-4.3.0-2.el7.x86_64 ovirt-vmconsole-1.0.6-2.el7.noarch ovirt-imageio-proxy-1.4.6-1.el7.noarch ovirt-engine-wildfly-overlay-14.0.1-3.el7.noarch ovirt-web-ui-1.5.0-1.el7.noarch ovirt-provider-ovn-1.2.19-1.el7.noarch ovirt-engine-dbscripts-4.3.0.2-1.el7.noarch ovirt-ansible-engine-setup-1.1.7-1.el7.noarch ovirt-engine-setup-plugin-ovirt-engine-4.3.0.4-1.el7.noarch ovirt-provider-ovn-driver-1.2.19-1.el7.noarch ovirt-engine-extension-aaa-jdbc-1.1.8-1.el7.noarch python2-ovirt-setup-lib-1.2.0-1.el7.noarch ovirt-host-deploy-java-1.8.0-1.el7.noarch ovirt-engine-tools-4.3.0.2-1.el7.noarch cockpit-machines-ovirt-176-4.el7.centos.noarch ovirt-engine-metrics-1.2.0.3-1.el7.noarch ovirt-imageio-proxy-setup-1.4.6-1.el7.noarch ovirt-vmconsole-host-1.0.6-2.el7.noarch ovirt-ansible-manageiq-1.1.13-1.el7.noarch ovirt-engine-wildfly-14.0.1-3.el7.x86_64 ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.0.4-1.el7.noarch ovirt-host-dependencies-4.3.0-2.el7.x86_64 ovirt-ansible-disaster-recovery-1.1.4-1.el7.noarch ovirt-engine-dwh-setup-4.3.0-1.el7.noarch ovirt-ansible-v2v-conversion-host-1.9.1-1.el7.noarch python2-ovirt-engine-lib-4.3.0.4-1.el7.noarch ovirt-release43-pre-4.3.0-0.7.rc4.el7.noarch

3 5

Sanlock volume corrupted on deployment
by Strahil 31 Jan '19

31 Jan '19

4 9

Changing DNS servers
by Chris Adams 31 Jan '19

31 Jan '19

I am trying to change the DNS servers on the hosts on a 4.1 cluster. It appears that oVirt has "learned" them somewhere. They are in /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt (and of course /etc/resolv.conf), but that file gets rewritten on a reboot (so if I change it they just change back). I can't find anything in the web UI to set/change DNS, but obviously they're stored somewhere. I see /var/lib/vdsm/persistence/netconf.<number>/nets/ovirtmgmt has the config; is it safe to directly edit that file? -- Chris Adams <cma(a)cmadams.net>

2 2

[ANN] oVirt 4.3.0 Fifth Release Candidate is now available for testing
by Sandro Bonazzola 31 Jan '19

31 Jan '19

The oVirt Project is pleased to announce the availability of the Fifth Release Candidate of oVirt 4.3.0, as of January 31th, 2019 This is pre-release software. This pre-release should not to be used in production. Please take a look at our community page[1] to learn how to ask questions and interact with developers and users. All issues or bugs should be reported via oVirt Bugzilla[2]. This update is the first release candidate of the 4.3.0 version. This release brings more than 130 enhancements and more than 450 bug fixes on top of oVirt 4.2 series. What's new in oVirt 4.3.0? * Q35 chipset, support booting using UEFI and Secure Boot * Skylake-server and AMD EPYC support * New smbus driver in windows guest tools * Improved support for v2v * OVA export / import of Templates * Full support for live migration of High Performance VMs * Microsoft Failover clustering support (SCSI Persistent Reservation) for Direct LUN disks * Hundreds of bug fixes on top of oVirt 4.2 series * New VM portal details page (see a preview here: https://imgur.com/a/ExINpci) * New Cluster upgrade UI * OVN security groups * IPv6 (static host addresses) * Support of Neutron from RDO OpenStack 13 as external network provider * Support of using Skydive from RDO OpenStack 14 as Tech Preview * Support for 3.6 and 4.0 data centers, clusters and hosts were removed * Now using PostgreSQL 10 * New metrics support using rsyslog instead of fluentd This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes draft [3] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available for CentOS 7 - oVirt Appliance for Fedora 28 (tech preview) is being delayed due to build issues with the build system. Additional Resources: * Read more about the oVirt 4.3.0 release highlights: http://www.ovirt.org/release/4.3.0/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] https://www.ovirt.org/community/ [2] https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt [3] http://www.ovirt.org/release/4.3.0/ [4] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

host NonResponsive after trying to update network MTUs
by Edward Berger 31 Jan '19

31 Jan '19

I have a problem host which also is the one I deployed a hyperconverged oVirt node-ng cluster from with the cockpit's hyperconverged installation wizard. When I realized after deploying that I hadn't set the MTUs correctly for the engine-mgmt, associated vlan and eno.2 device and also for my infiniband interface ib0, I went in and tried to set them to new values 9000 and 65520 it got into some kind of hung state. The engine task window shows a task in "executing" and a never ending spinning widget "Handing non responsive Host track00..." I can tried updating the hosts /etc/sysconfig/network-scripts by hand. I've tried every combination of the engines set host in maintenance mode , "sync networks" "refresh host capabilities" activating and rebooting, but I'm still stuck with an unresponsive host. I had another host that also failed but it allowed me to put it into maintenance mode and then remove it from the cluster and "add new" it back and it was happy. This one won't let me remove it because its serving the gluster volume mount point, even though I did give the mount options for the 2nd and 3rd backup volume servers. I'd appreciate any help restoring it to proper working order. I'm attaching the gzip'd engine log.

2 5

Failed to configure management network on host Local due to setup networks failure.
by rafonseca＠gmail.com 31 Jan '19

31 Jan '19

Friends, I have install oVirt but appears failed. And interfaces stay not available (LLPD) and not sync. I try install on CentOS and Scientific Linux same problems

2 1

ovirt-sdk_external network add
by ada per 31 Jan '19

31 Jan '19

Hello everyone, I have the following script, i ve been looking in ovirt-sdk but i cannot seem to find the proper way of adding an external provider network under ovirt-ovn I manage to add logical networks and vlans but no luck in external provider. Any advice is appreciated network = networks_service.add( network=types.Network( name='ext_net', description='Testing network', data_center=types.DataCenter( name='Default' ), usages=[types.NetworkUsage.VM], external_provider='ovirt-provider-ovn', -->i know this part is wrong what is it supposed to be called? ),

2 10

Changing CPU type in oVirt Cluster 4.2
by Juhani Rautiainen 31 Jan '19

31 Jan '19

Hi! I made two node oVirt cluster with self hosted engine. It's running top of oVirt Node 4.2. Hardware is built on AMD Epyc processors. I just noticed that "Clusters"-view warns that I have AMD Opteron G3 as CPU type and it will not supported in the future. Didn't check this during installation so it probably came automatically. But question is how can I change this? I get warning 'Error while executing action: Cannot change Cluster CPU to higher CPU type when there are active Hosts with lower CPU type.'. I saw old thread from 2016 where it was suggested that you change value straight to db. Put that seems bit dangerous. Is there any other way to solve this problem? Thanks, Juhani -- Juhani Rautiainen jrauti(a)iki.fi

1 0

Deploying single instance - error
by Leo David 31 Jan '19

31 Jan '19

Hello Everyone, Using version 4.2.8, ( ovirt-node-ng-installer-4.2.0-2019012606.el7.iso ) for deploying one node instance by following from within CockpitUI seems not to be possible. Here's the generated inventory ( i've specified "jbod" in the wizard ): #gdeploy configuration generated by cockpit-gluster plugin [hosts] 192.168.80.191 [script1:192.168.80.191] action=execute ignore_script_errors=no file=/usr/share/gdeploy/scripts/grafton-sanity-check.sh -d sdb -h 192.168.80.191 [disktype] jbod [service1] action=enable service=chronyd [service2] action=restart service=chronyd [shell2] action=execute command=vdsm-tool configure --force [script3] action=execute file=/usr/share/gdeploy/scripts/blacklist_all_disks.sh ignore_script_errors=no [pv1:192.168.80.191] action=create devices=sdb ignore_pv_errors=no [vg1:192.168.80.191] action=create vgname=gluster_vg_sdb pvname=sdb ignore_vg_errors=no [lv1:192.168.80.191] action=create lvname=gluster_lv_engine ignore_lv_errors=no vgname=gluster_vg_sdb mount=/gluster_bricks/engine size=230GB lvtype=thick [selinux] yes [service3] action=restart service=glusterd slice_setup=yes [firewalld] action=add ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp,54322/tcp services=glusterfs [script2] action=execute file=/usr/share/gdeploy/scripts/disable-gluster-hooks.sh [shell3] action=execute command=usermod -a -G gluster qemu [volume1] action=create volname=engine transport=tcp key=storage.owner-uid,storage.owner-gid,features.shard,performance.low-prio-threads,performance.strict-o-direct,network.remote-dio,network.ping-timeout,user.cifs,nfs.disable,performance.quick-read,performance.read-ahead,performance.io-cache,cluster.eager-lock value=36,36,on,32,on,off,30,off,on,off,off,off,enable brick_dirs=192.168.80.191:/gluster_bricks/engine/engine ignore_volume_errors=no It does not get to finish, throwing the following error: PLAY [gluster_servers] ********************************************************* TASK [Create volume group on the disks] **************************************** changed: [192.168.80.191] => (item={u'brick': u'/dev/sdb', u'vg': u'gluster_vg_sdb'}) PLAY RECAP ********************************************************************* 192.168.80.191 : ok=1 changed=1 unreachable=0 failed=0 *Error: Section diskcount not found in the configuration file* Any thoughts ? -- Best regards, Leo David

5 10

Re: Error in SeaBIOS
by Eyal Edri 31 Jan '19

31 Jan '19

On Thu, Jan 31, 2019, 05:45 LIONS TECNOLOGIA <lionstecnologiati(a)gmail.com wrote: > Good night! > > I would like to have a help, I am starting this journey in virtualization > and I am having difficulty, I installed the ovirt but when I install a > virtual machine to test, it happens that it is black screen as described in > the attachment. > Anyone who has a little more experience could help? > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCLWGIWZPBXOX… >

1 0

VDO volume - storage domain size
by Leo David 31 Jan '19

31 Jan '19

Hello Everyone, I'm I trying to benefit of vdo capabilities, but it seems that I don;t have too much luck with this. I have both created a vdo based gluster volume usign both following methods: - at hyperconverged wizard cluster setup by enabling compression per device - after installation, by creating a new gluster volume and enabling compression In both cases, I end up with a real device size gluster volume/storage domain, although the vdo's appear in node's cockpit UI as being 10 times then physical device. ie: 3 nodes, each having 1 x 900GB ssd device, turns into 9t device vdo device per host, but the storage domain (gluster replica 3 ) ends up as being 900GB . Am I missing something , or maybe doing something wrong? Thank you very much ! Leo Best regards, Leo David

2 3

Roles and Permissions and Inheritance
by Brian Wilson 30 Jan '19

30 Jan '19

Is there a way to prevent Roles Assigned to Groups on Objects to only apply to where it is set? Basically looking for a way to do what we had done in VMWare which involved using the do not propagate permission setting. be able Seems to me that right now there is no way to set this so if i give access to something at the top level of a DC those accesses wlll overide if i then explcitly set another role and permission on an object underneath Lets take as a concrete example the ovirtmgmt network. I do not want users in the engine to be able to place VMs on this (but i want the Superusers to be able to still) How can i accomplish this with the way roles and permissions work with Ovirt? thanks! Brian

3 4

Ovirt configuration
by matteofedeli97＠gmail.com 30 Jan '19

30 Jan '19

Hi at all, I'm installing and configuring a ovirt system in hyperconverged mode and I have some doubt about storage. server 1: 1TB, 1TB server 2: 1TB, 1TB server 3: 1TB, 1TB With this hdd configuration I want on the first drive install ovirt node, vmstore and engine volume, on the second another volume for data. The first in Replica 3 and the second distribuited... It's possible? During installation of ovirt node in the automatic partitioning should I select both hard disk? In the cockpit wizard in this step, https://www.ovirt.org/images/uarwo42-gdeploy-3.png?1519899774, to recover the device name i use pvdisplay -m?

3 5

vSphere Admin is trying to setup an oVirt-Cluster...
by ralph＠gottschalkson.eu 30 Jan '19

30 Jan '19

and is failing .. of course. I get access to 3 HP DL380 G7 with a lot of RAM and a lot of HDDs an 10 GBE NICs. Which VLANs I have to create? The HDDs have not to be in a RAID, they should be single instances for GlusterFS, so that GlusterFS Handles the "Bricks (=Single HHDs?)"? Or use a SAN shared Device, which is also there for simplyfienig the (first) setup What to do with the downloaded oVirt-ISO Image? In vSphere I install ESXi on the nodes, Connect the NICs to the right VLANs. Install VCSA as VM, adding the nodes and so on.. I there an good Instruction out there which tanslates VMWare terms and concepts into oVirts terms so that I can follow the step-by-step instuction? Or ist the setup done by an (magic) Ansible script?

2 1

4.3.0 rc2 cannot mount glusterfs volumes on ovirt node ng
by Jorick Astrego 30 Jan '19

30 Jan '19

Hi, We're having problems mounting the preexisting 3.12 glusterfs storage domains in ovirt node ng 4.3.0 rc2. Getting There are no iptables blocks on the storage network, the ip's are pingable bothe ways. I can telnet to the glusterfs ports and I see no messages in the logs of the glusterfs servers. When I try the mount command manually it hangs for ever: /usr/bin/mount -t glusterfs -o backup-volfile-servers=*.*.*.*:*.*.*.* *.*.*.*:/sdd8 /mnt/temp I haven't submitted a bug yet.... from supervdsm.log MainProcess|jsonrpc/2::DEBUG::2019-01-25 13:42:45,282::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) call volumeInfo with (u'sdd8', u'*.*.*.*') {} MainProcess|jsonrpc/2::DEBUG::2019-01-25 13:42:45,282::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.* sdd8 --xml (cwd None) MainProcess|jsonrpc/2::DEBUG::2019-01-25 13:44:45,399::commands::219::root::(execCmd) FAILED: <err> = ''; <rc> = 1 MainProcess|jsonrpc/2::DEBUG::2019-01-25 13:44:45,399::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 120 seconds (max pending: 2) MainProcess|jsonrpc/2::ERROR::2019-01-25 13:44:45,399::supervdsm_server::104::SuperVdsm.ServerCallback::(wrapper) Error in volumeInfo Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 102, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 529, in volumeInfo xmltree = _execGlusterXml(command) File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 131, in _execGlusterXml return _getTree(rc, out, err) File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 112, in _getTree raise ge.GlusterCmdExecFailedException(rc, out, err) GlusterCmdExecFailedException: Command execution failed error: E r r o r : R e q u e s t t i m e d o u t return code: 1 MainProcess|jsonrpc/2::DEBUG::2019-01-25 13:44:45,400::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) call mount with (<vdsm.supervdsm_server._SuperVdsm object at 0x7f6eb8d0a2d0>, u'*.*.*.*:/sdd8', u'/rhev/data-center/mnt/glusterSD/*.*.*.*:_sdd8') {'vfstype': u'glusterfs', 'mntOpts': u'backup-volfile-servers=*.*.*.*:*.*.*.*', 'cgroup': 'vdsm-glusterfs'} MainProcess|jsonrpc/2::DEBUG::2019-01-25 13:44:45,400::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount -t glusterfs -o backup-volfile-servers=*.*.*.*:*.*.*.* *.*.*.*:/sdd8 /rhev/data-center/mnt/glusterSD/*.*.*.*:_sdd8 (cwd None) MainProcess|jsonrpc/0::DEBUG::2019-01-25 13:45:02,884::commands::219::root::(execCmd) FAILED: <err> = 'Running scope as unit run-38676.scope.\nMount failed. Please check the log file for more details.\n'; <rc> = 1 MainProcess|jsonrpc/0::ERROR::2019-01-25 13:45:02,884::supervdsm_server::104::SuperVdsm.ServerCallback::(wrapper) Error in mount Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 102, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 144, in mount cgroup=cgroup) File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (1, ';Running scope as unit run-38676.scope.\nMount failed. Please check the log file for more details.\n') MainProcess|jsonrpc/0::DEBUG::2019-01-25 13:45:02,894::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) call volumeInfo with (u'ssd9', u'*.*.*.*') {} MainProcess|jsonrpc/0::DEBUG::2019-01-25 13:45:02,894::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.* ssd9 --xml (cwd None) from vdsm.log 2019-01-25 13:46:03,519+0100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/2 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'6b6b7899-c82b-4417-b453-0b3b0ac11deb', u'connection': u'*.*.*.*:ssd4', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.*:ssd5', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'40d191b0-b7f8-48f9-bf6f-327275f51fef', u'connection': u'*.*.*.*:ssd6', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'26fbd2d6-6f25-4520-ab7f-15e9001f07b9', u'connection': u'*.*.*.*:/hdd2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'f11fed97-513a-4a10-b85c-2afe68f42608', u'connection': u'*.*.*.*:/ssd3', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'f20b8691-528e-4e38-89ad-1e27684dee8b', u'connection': u'*.*.*.*:/sdd8', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'188e71dc-3d81-43d3-b930-238a4c6711e6', u'connection': u'*.*.*.*:/ssd9', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'domainType': 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'581e2ad3-0682-4d44-95b4-bdc088b45f66'} at 0x7f9be815c850> timeout=60, duration=1260.00 at 0x7f9be815ca10> task#=98 at 0x7f9be83bb750>, traceback: File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run self._execute_task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1103, in connectStorageServer connectionParams) File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 72, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File: "<string>", line 2, in connectStorageServer File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2410, in connectStorageServer conObj.connect() from messages: Jan 25 13:49:07 node9 vdsm[31968]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'6b6b7899-c82b-4417-b453-0b3b0ac11deb', u'connection': u'192.168.99.15:ssd4', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.*:ssd5', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'40d191b0-b7f8-48f9-bf6f-327275f51fef', u'connection': u'*.*.*.*:ssd6', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'26fbd2d6-6f25-4520-ab7f-15e9001f07b9', u'connection': u'*.*.*.*:/hdd2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'f11fed97-513a-4a10-b85c-2afe68f42608', u'connection': u'*.*.*.*:/ssd3', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'f20b8691-528e-4e38-89ad-1e27684dee8b', u'connection': u'*.*.*.*:/sdd8', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': u'188e71dc-3d81-43d3-b930-238a4c6711e6', u'connection': u'*.*.*.*:/ssd9', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'domainType': 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'918f4d06-ca89-4ec0-a396-3407f1bdb8f9'} at 0x7f9be82ff250> timeout=60, duration=840.00 at 0x7f9be82ffa50> task#=93 at 0x7f9be83bba10>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1103, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 72, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2410, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info(a)netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ----------------

5 8

Administration Portal High Availability
by ernestclyde＠sav25.com 30 Jan '19

30 Jan '19

Hello, Currently i am finding some Documentation or "how to" setup High Availability of oVirt Engine's Administration Portal.

3 3

Host Only Network
by Brian Wilson 29 Jan '19

29 Jan '19

What is the proper way to provide networks without an egress of the host? In VMware i would take a portgroup and just not associate it with a vnic. In ovirt that doesnt seem possible as the network doesnt seem to function unless i move to a physical nic under the "Setup Host Networks" options. The Goal here is to provide a network for VMs to use on the single host only no traffic would need to leave the host but would need to have VLan IDs and segmentation through them on the host

4 4

Single Node Deployment - Self-Hosted-Engine
by Brian Wilson 29 Jan '19

29 Jan '19

I have a question regarding installing a single node and then running the self-hosted engine on it. I would like to keep this as simple as possible and am running into an issue with where to tell the installer to place the engine. It asks me to choose a shared storage technology(gfs,nfs, iscsi) and i dont want to rely on anything outside of this single box to provide storage. I would like to add another volume (LVM) as a local storage domain to be used for this as well as all other VMs to be run on this all in one box. Is there some secret sauce to getting this installer to choose a local location rather than a shared one?

4 6

[ANN] oVirt 4.3.0 Fourth Release Candidate is now available for testing
by Sandro Bonazzola 29 Jan '19

29 Jan '19

The oVirt Project is pleased to announce the availability of the Fourth Release Candidate of oVirt 4.3.0, as of January 29th, 2019 This is pre-release software. This pre-release should not to be used in production. Please take a look at our community page[1] to learn how to ask questions and interact with developers and users. All issues or bugs should be reported via oVirt Bugzilla[2]. This update is the first release candidate of the 4.3.0 version. This release brings more than 130 enhancements and more than 450 bug fixes on top of oVirt 4.2 series. What's new in oVirt 4.3.0? * Q35 chipset, support booting using UEFI and Secure Boot * Skylake-server and AMD EPYC support * New smbus driver in windows guest tools * Improved support for v2v * OVA export / import of Templates * Full support for live migration of High Performance VMs * Microsoft Failover clustering support (SCSI Persistent Reservation) for Direct LUN disks * Hundreds of bug fixes on top of oVirt 4.2 series * New VM portal details page (see a preview here: https://imgur.com/a/ExINpci) * New Cluster upgrade UI * OVN security groups * IPv6 (static host addresses) * Support of Neutron from RDO OpenStack 13 as external network provider * Support of using Skydive from RDO OpenStack 14 as Tech Preview * Support for 3.6 and 4.0 data centers, clusters and hosts were removed * Now using PostgreSQL 10 * New metrics support using rsyslog instead of fluentd This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes draft [3] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available for CentOS 7 - oVirt Node NG is already available for both CentOS 7 and Fedora 28 (tech preview). - oVirt Appliance for Fedora 28 (tech preview) is being delayed due to build issues with the build system. Additional Resources: * Read more about the oVirt 4.3.0 release highlights: http://www.ovirt.org/release/4.3.0/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] https://www.ovirt.org/community/ [2] https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt [3] http://www.ovirt.org/release/4.3.0/ [4] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

Planned infra outage - Saturday February 2
by Evgheni Dereveanchin 29 Jan '19

29 Jan '19

Hi everyone, There's a maintenance window planned for this Saturday, February 2 2019 to upgrade network equipment in the PHX datacenter. The change will be implemented during afternoon hours and several oVirt infra services may be unreachable for short periods of time, specifically: * package repositories * Jenkins CI * glance image repo If you have problems accessing project resources during this period or see CI failures please re-try in a couple of hours. The project website and Gerrit code review are not affected by this outage and will continue to function normally. -- Regards, Evgheni Dereveanchin

1 0

Clearing asynchronous task Unknown
by nicolas＠devels.es 29 Jan '19

29 Jan '19

Hi, We're running oVirt 4.1.9 (I know there's a new version, we can't upgrade until [1] is implemented). The thing is that since some days we're having an event that floods our event list: Clearing asynchronous task Unknown that started at Tue Jan 22 14:33:17 WET 2019 The event shows up every minute. We tried restarting the ovirt-engine, but after some time it starts flooding again. No pending tasks in the task list. How can I check what is happening and how to solve it? Thanks. [1]: https://github.com/oVirt/ovirt-web-ui/issues/490

2 2