- Users - oVirt List Archives

Sigh... still vdsmd memory leak
by Chris Adams 03 Dec '19

03 Dec '19

I've seen vdsmd leak memory (RSS increasing) for a while, and never gotten anywhere with diagnosing it. I upgraded a small setup (2 hosts with hosted engine and iSCSI SAN) last week to 4.3.7 plus latest CentOS 7.7 updates, and I am still seeing this. Currently, there are only 3 active VMs, plus the engine, all running on one of the two node (which is also the SPM), and vdsmd on that node is up to 450M RSS (6 days after upgrading), which is up almost 100M in the last 24 hours. On the other node, vdsmd RSS has not changed in the last 24 hours (83M). I have a script watching the RSS on the growing node, and it's a pretty steady increase: 07:59:59.735 452080 +28 08:00:03.361 452104 +24 08:00:04.852 452112 +8 08:00:21.088 452116 +4 08:00:25.456 452120 +4 08:00:25.753 452124 +4 08:00:34.219 452132 +8 08:00:35.866 452136 +4 08:00:44.360 452140 +4 08:00:51.213 452144 +4 08:00:57.046 452152 +8 08:00:59.855 452160 +8 08:01:01.461 452164 +4 08:01:04.615 452168 +4 08:01:10.544 452172 +4 08:01:11.003 452180 +8 08:01:13.826 452184 +4 08:01:14.737 452188 +4 08:01:15.002 452192 +4 08:01:21.350 452196 +4 08:01:29.870 452200 +4 08:01:34.995 452208 +8 This is a problem I have in multiple oVirt setups - don't know if it is just something about how I installed or what, but I'd really like to work on getting this figured out and fixed. -- Chris Adams <cma(a)cmadams.net>

1 0

Scenario: Ovirt-engine on hardware and recovery
by rwebb＠ropeguru.com 03 Dec '19

03 Dec '19

So thinking about my setup, I am thinking through different failure scenarios. So lets say I have a small physical server with 8 cores and 16GB RAM and I install Centos 7 and ovirt-engine on bare metal. This would also be the same scenario is the engine were on a VM. I run into a major hardware issue and completely lose the engine. How does one recover the cluster setup and not have to start from scratch by having to rebuild all the nodes? Can the engine just be rebuilt and the ovirt nodes be imported? This scenario is based on the nodes being built from the ovirt node iso.

3 2

Re: Windows VirtIO drivers
by Gal Zaidman 03 Dec '19

03 Dec '19

Going back to the user list, just in case it will be relevant to others On Tue, Dec 3, 2019 at 3:02 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> wrote: > Hi Gal, > > > > Thank you so much for replying, can you please a steps to do that. Right > now what I am doing is: > > > > 1. Installed Windows 2016 using IDE interface, but performance is very > bad and it’ really hard to install any further .exe into it. > > I never used the IDE interface so I can't say from personal experience, but I don't think that should be the behavior, please open a bug on that so that we can examin. > > 1. > 2. Other option I see attaching the floppy drive when we use “Run > Once”, but problem is when pointing to floppy drive while loading drivers. > It says no drivers detected. > > Using Run Once is the preferred option, see https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/ht… Make sure that you attach the latest virtio-win vdf (the link I sent earlier should be ok) and select the relevant drivers, if there is a problem then please open a bug on virtio-win project and send me the url so I will track it, thanks. > > > Please share let me know the best way to perform this action as need to > deliver the Ovirt infra to one of my customer and really stuck on this > driver installation. > > > > Appreciate your quick response.!! > > > > Vijay Sachdeva > > > > > > *From: *Gal Zaidman <gzaidman(a)redhat.com> > *Date: *Tuesday, 3 December 2019 at 6:22 PM > *To: *Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> > *Subject: *Re: [ovirt-users] Windows VirtIO drivers > > > > HI Vijay, > > > > There a package on fedora: > > virtio-win-0.1.173-2 > > which contains an MSI installer for the drivers and the ovirt agent (which > is disabled on a normal installation, so please enable it when you install). > > > > This installer should replace the ovirt-wgt that we have currently on the > next versions, > > and it is new, so please give us feedback once you used it. > > > > P.S > > the installer installs the drivers normally so if the regular drivers from > virtio-win fail to install, then using the installer will not change much > probably, and it means that you have a deeper problem. > > In case the installation fails then please generate logs but running the > installer through cmd.exe "msiexec /i {installer} /l*v {path to log}" ( > https://www.advancedinstaller.com/user-guide/qa-log.html) > > and open a bug/reply here and I will try and help > > > > Thanks > > > > On Tue, Dec 3, 2019 at 2:11 PM Vijay Sachdeva <vijay.sachdeva(a)indiqus.com> > wrote: > > Dear Team, > > > > How can we install virtIO windows drivers to Windows 2016. As of now I > tried to attach floppy drive with virtio-win.vfd image but while installing > drivers didn’t get loaded. > > > > Please guide or let me know the proper way to install drivers. > > > > > > Vijay Sachdeva > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XBYY54K62BYJ… > >

1 0

Disk move succeed but didn't move content
by Juan Pablo Lorier 03 Dec '19

03 Dec '19

Hi, I've a fresh new install of ovirt 4.3 and tried to import an gluster vmstore. I managed to import via NFS the former data domain. The problem is that when I moved the disks of the vms to the new ISCSI data domain, I got a warning that sparse disk type will be converted to qcow2 disks, and after accepting, the disks were moved with no error. The problem is that the disks now figure as <1Gb size instead of the original size and thus, the vms fail to start. Is there any way to recover those disks? I have no backup of the vms :-( Regards

3 4

Current status of Ceph support in oVirt (2019)?
by victorhooi＠yahoo.com 03 Dec '19

03 Dec '19

Hi, I currently have a 3-node HA cluster running Proxmox (with integrated Ceph). oVirt looks pretty neat, however, and I'm excited to check it out. One of the things I love about Proxmox is the integrated Ceph support. I saw on the mailing lists that there is some talk of Ceph support earlier, but it was via OpenStack/Cinder. What exactly does this mean? 1. Does this require you to install OpenStack, or will a vanilla Ceph installation work? 2. Is it possible to deploy Ceph on the same nodes that run oVirt? (i.e. is a 3-node oVirt + Ceph cluster possible?) 3. Is there any monitoring/management of Ceph from within oVirt? (Guessing no?) 4. Are all the normal VM features working yet, or is this planned? 5. Is making Ceph a first-class citizen (like Gluster) on oVirt on the roadmap? Thanks, Victor https://www.reddit.com/r/ovirt/comments/ci38zp/ceph_rbd_support_in_ovirt_fo…

3 3

Windows VirtIO drivers
by Vijay Sachdeva 03 Dec '19

03 Dec '19

Dear Team, How can we install virtIO windows drivers to Windows 2016. As of now I tried to attach floppy drive with virtio-win.vfd image but while installing drivers didn’t get loaded. Please guide or let me know the proper way to install drivers. Vijay Sachdeva

1 0

oVirt nodes not responding
by Tim Herrmann 03 Dec '19

03 Dec '19

Hi everyone, we have an oVirt-Cluster with 5 nodes and 3 of them provide the storage with GlusterFS and replica 3. The cluster is running 87 VMs and has 9TB storage where 4TB is in use. The version of the oVirt-Engine is 4.1.8.2 and GlusterFS is 3.8.15. The servers are running in a HP bladecenter and are connected with 10GBit to each other. Actually we have some problems that all ovirt nodes periodically won't respond in the cluster with the following error messages in the oVirt webinterface: VDSM glustervirt05 command GetGlusterVolumeHealInfoVDS failed: Message timeout which can be caused by communication issues Host glustervirt05 is not responding. It will stay in Connecting state for a grace period of 68 seconds and after that an attempt to fence the host will be issued. Host glustervirt05 does not enforce SELinux. Current status: PERMISSIVE Executing power management status on Host glustervirt05 using Proxy Host glustervirt02 and Fence Agent ilo4:xxx.xxx.xxx.xxx. Manually synced the storage devices from host glustervirt05 Status of host glustervirt05 was set to Up. In the vdsm logfile I can find the following message: 2019-11-26 11:18:22,909+0100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {u'volumeName': u'data'}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.healInfo', 'id': u'2e86ed2c- 3e79-42c1-a7e4-c09bfbfc7794'} at 0x7fb938373190> timeout=60, duration=180 at 0x316a6d0> task#=2859802 at 0x1b70dd0> (executor:351) And I figured out, that the gluster heal info command takes very long: [root@glustervirt01 ~]# time gluster volume heal data info Brick glustervirt01:/gluster/data/brick1 Status: Connected Number of entries: 0 Brick glustervirt02:/gluster/data/brick1 Status: Connected Number of entries: 0 Brick glustervirt03:/gluster/data/brick2 Status: Connected Number of entries: 0 real 3m3.626s user 0m0.593s sys 0m0.559s A strange behavier is also that there is one virtual machine (a postgresql database) which stops running unexpectedly every one or two days ... The only thing that has been changed on the vm in the least past was a resize of the disk. VM replication-zabbix is down with error. Exit message: Lost connection with qemu process. And when we add or delete a larger disk with approximately 100GB in glusterfs, the glusterfs cluster freaks out won't respond anymore. This also results in paused VMs ... Has anyone an idea what could cause such problems?

2 1

Re: Scenario: Ovirt-engine on hardware and recovery
by Strahil 03 Dec '19

03 Dec '19

As this is not a VM, you won't have to worry about putting 1 host in maintenance. I make backups lile this: engine-backup --mode=backup --scope=all --file=/root/engine-bkp-empty-ovirt1-ver4.3.7-rc4--2019-11-30 --log=/var/log/engine-backup.log I'm specifying the host that will be used for restore, which is needed when the engine is VM. In your bare-metal case - you just need to recover the OS/harfware and restore from backup. Best Regards, Strahil NikolovOn Dec 3, 2019 04:36, rwebb(a)ropeguru.com wrote: > > So thinking about my setup, I am thinking through different failure scenarios. > > So lets say I have a small physical server with 8 cores and 16GB RAM and I install Centos 7 and ovirt-engine on bare metal. This would also be the same scenario is the engine were on a VM. > > I run into a major hardware issue and completely lose the engine. How does one recover the cluster setup and not have to start from scratch by having to rebuild all the nodes? Can the engine just be rebuilt and the ovirt nodes be imported? > > This scenario is based on the nodes being built from the ovirt node iso. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XBFYDGZNCT4VL…

1 0

Re: host failed to attach one of the storage domains attached to it
by Strahil 03 Dec '19

03 Dec '19

Hi Daniel, The hosts go to unresponsive mode as one of the gluster volumes cannot be mounted. Verify that all host can mount (over /mnt) all gluster volumes. If they do, just set the hosts into maintenance and then activate them. Best Regards, Strahil NikolovOn Dec 2, 2019 12:03, Daniel Menzel <daniel.menzel(a)menzel-it.net> wrote: > > Dear all, > > after a GlusterFS reboot of two servers out of three (and thus a loss of quorum) by an admin we've got strange problems: > > The hosted engine and one export work perfectly fine. (Same three servers!) > Another export (main one) seems to be fine within GlusterFS itself. But: When we activate this domain most of the hosts go into "non operational" with the subject's error message. As soon as we deactivate this domain all those hosts come back online. Strange thing: The GlusterFS export seems to be mounted on the SPM. > > Does anyone know what could have happened and how to fix that? > > Kind regards > Daniel

1 0

Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
by Strahil 03 Dec '19

03 Dec '19

Hi Jiffin, Thanks for the info. As I'm now on Gluster v7 , I hope it won't happen again. It's nice to know it got fixed. Best Regards, Strahil NikolovOn Dec 2, 2019 11:32, Jiffin Thottan <jthottan(a)redhat.com> wrote: > > Hi Krutika, > > Apparently, in context acl info got corrupted see brick logs > > [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: > client: > CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > gfid: be318638-e8a0-4c6d-977d-7a937aa84806, > req(uid:107,gid:107,perm:1,ngrps:4), > ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) > [Permission denied] > > which resulted in the situation. There was one bug similar was reported https://bugzilla.redhat.com/show_bug.cgi?id=1668286 and > > it got fixed in 6.6 release IMO https://review.gluster.org/#/c/glusterfs/+/23233/. But here he mentioned he saw the issue when > > he upgraded from 6.5 to 6.6 > > One way to workaround is to perform a dummy setfacl(preferably using root) on the corrupted files which will forcefully fetch the acl > > info again from backend and update the context. Another approach to restart brick process(kill and vol start force) > > Regards, > Jiffin > > ----- Original Message ----- > From: "Krutika Dhananjay" <kdhananj(a)redhat.com> > To: "Strahil Nikolov" <hunter86_bg(a)yahoo.com>, "Jiffin Thottan" <jthottan(a)redhat.com>, "raghavendra talur" <rtalur(a)redhat.com> > Cc: "Nir Soffer" <nsoffer(a)redhat.com>, "Rafi Kavungal Chundattu Parambil" <rkavunga(a)redhat.com>, "users" <users(a)ovirt.org>, "gluster-user" <gluster-users(a)gluster.org> > Sent: Monday, December 2, 2019 11:48:22 AM > Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing > > Sorry about the late response. > > I looked at the logs. These errors are originating from posix-acl > translator - > > > > *[2019-11-17 07:55:47.090065] E [MSGID: 115050] > [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496: > LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6 > (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6), > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > error-xlator: data_fast-access-control [Permission denied][2019-11-17 > 07:55:47.090174] I [MSGID: 139001] > [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > gfid: be318638-e8a0-4c6d-977d-7a937aa84806, > req(uid:36,gid:36,perm:1,ngrps:3), > ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) > [Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050] > [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497: > LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7 > (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7), > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > error-xlator: data_fast-access-control [Permission denied][2019-11-17 > 07:55:47.090299] I [MSGID: 139001] > [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: > client: > CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, > gfid: be318638-e8a0-4c6d-977d-7a937aa84806, > req(uid:36,gid:36,perm:1,ngrps:3), > ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) > [Permission denied]* > > Jiffin/Raghavendra Talur, > Can you help? > > -Krutika > > On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> > wrote: > > > Hi Nir,All, > > > > it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached > > screenshot of oVirt running on v7 gluster). > > It seems strange that both my serious issues with oVirt are related to > > gluster issue (1st gluster v3 to v5 migration and now this one). > > > > I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all > > nodes. > > Now both Engine and all my VMs are back online - so if you hit issues with > > 6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before > > deciding to wipe everything. > > > > @Krutika, > > > > I guess you will ask for the logs, so let's switch to gluster-users about > > this one ? > > > > Best Regards, > > Strahil Nikolov > > > > В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov < > > hunter86_bg(a)yahoo.com> написа: > > > > > > Hi Krutika, > > > > I have enabled TRACE log level for the volume data_fast, > > > > but the issue is not much clear: > > FUSE reports: > > > > [2019-11-25 21:31:53.478130] I [MSGID: 133022] > > [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of > > gfid=6d9ed2e5-d4f2-4749-839b-2f1 > > 3a68ed472 from backend > > [2019-11-25 21:32:43.564694] W [MSGID: 114031] > > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: > > remote operation failed. Path:

1 0