Disk move succeed but didn't move content
by Juan Pablo Lorier
Hi,
I've a fresh new install of ovirt 4.3 and tried to import an gluster
vmstore. I managed to import via NFS the former data domain. The problem
is that when I moved the disks of the vms to the new ISCSI data domain,
I got a warning that sparse disk type will be converted to qcow2 disks,
and after accepting, the disks were moved with no error.
The problem is that the disks now figure as <1Gb size instead of the
original size and thus, the vms fail to start.
Is there any way to recover those disks? I have no backup of the vms :-(
Regards
4 years, 4 months
Current status of Ceph support in oVirt (2019)?
by victorhooi＠yahoo.com
Hi,
I currently have a 3-node HA cluster running Proxmox (with integrated Ceph). oVirt looks pretty neat, however, and I'm excited to check it out.
One of the things I love about Proxmox is the integrated Ceph support.
I saw on the mailing lists that there is some talk of Ceph support earlier, but it was via OpenStack/Cinder. What exactly does this mean?
1. Does this require you to install OpenStack, or will a vanilla Ceph installation work?
2. Is it possible to deploy Ceph on the same nodes that run oVirt? (i.e. is a 3-node oVirt + Ceph cluster possible?)
3. Is there any monitoring/management of Ceph from within oVirt? (Guessing no?)
4. Are all the normal VM features working yet, or is this planned?
5. Is making Ceph a first-class citizen (like Gluster) on oVirt on the roadmap?
Thanks,
Victor
https://www.reddit.com/r/ovirt/comments/ci38zp/ceph_rbd_support_in_ovirt_...
4 years, 4 months
oVirt nodes not responding
by Tim Herrmann
Hi everyone,
we have an oVirt-Cluster with 5 nodes and 3 of them provide the storage
with GlusterFS and replica 3.
The cluster is running 87 VMs and has 9TB storage where 4TB is in use.
The version of the oVirt-Engine is 4.1.8.2 and GlusterFS is 3.8.15.
The servers are running in a HP bladecenter and are connected with 10GBit
to each other.
Actually we have some problems that all ovirt nodes periodically won't
respond in the cluster with the following error messages in the oVirt
webinterface:
VDSM glustervirt05 command GetGlusterVolumeHealInfoVDS failed: Message
timeout which can be caused by communication issues
Host glustervirt05 is not responding. It will stay in Connecting state for
a grace period of 68 seconds and after that an attempt to fence the host
will be issued.
Host glustervirt05 does not enforce SELinux. Current status: PERMISSIVE
Executing power management status on Host glustervirt05 using Proxy Host
glustervirt02 and Fence Agent ilo4:xxx.xxx.xxx.xxx.
Manually synced the storage devices from host glustervirt05
Status of host glustervirt05 was set to Up.
In the vdsm logfile I can find the following message:
2019-11-26 11:18:22,909+0100 WARN (vdsm.Scheduler) [Executor] Worker
blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params':
{u'volumeName': u'data'}, 'jsonrpc': '2.0', 'method':
u'GlusterVolume.healInfo', 'id': u'2e86ed2c-
3e79-42c1-a7e4-c09bfbfc7794'} at 0x7fb938373190> timeout=60, duration=180
at 0x316a6d0> task#=2859802 at 0x1b70dd0> (executor:351)
And I figured out, that the gluster heal info command takes very long:
[root@glustervirt01 ~]# time gluster volume heal data info
Brick glustervirt01:/gluster/data/brick1
Status: Connected
Number of entries: 0
Brick glustervirt02:/gluster/data/brick1
Status: Connected
Number of entries: 0
Brick glustervirt03:/gluster/data/brick2
Status: Connected
Number of entries: 0
real 3m3.626s
user 0m0.593s
sys 0m0.559s
A strange behavier is also that there is one virtual machine (a postgresql
database) which stops running unexpectedly every one or two days ...
The only thing that has been changed on the vm in the least past was a
resize of the disk.
VM replication-zabbix is down with error. Exit message: Lost connection
with qemu process.
And when we add or delete a larger disk with approximately 100GB in
glusterfs, the glusterfs cluster freaks out won't respond anymore.
This also results in paused VMs ...
Has anyone an idea what could cause such problems?
4 years, 4 months
Overt Networking VM not pinging
by Vijay Sachdeva
Dear Community,
I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working.
Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem?
Any help would be highly appreciated.
Thanks
Vijay Sachdeva
Senior Manager – Service Delivery
IndiQus Technologies
O +91 11 4055 1411 | M +91 8826699409
www.indiqus.com
4 years, 4 months
hyperconverged single node with SSD cache fails gluster creation
by thomas＠hoberg.net
I am seeing more success than failures at creating single and triple node hyperconverged setups after some weeks of experimentation so I am branching out to additional features: In this case the ability to use SSDs as cache media for hard disks.
I tried first with a single node that combined caching and compression and that fails during the creation of LVMs.
I tried again without the VDO compression, but actually the results where identical whilst VDO compression but without the LV cache worked ok.
I tried various combinations, using less space etc., but the results are always the same and unfortunately rather cryptic (substituted the physical disk label with {disklabel}):
TASK [gluster.infra/roles/backend_setup : Extend volume group] *****************
failed: [{hostname}] (item={u'vgname': u'gluster_vg_{disklabel}p1', u'cachethinpoolname': u'gluster_thinpool_gluster_vg_{disklabel}p1', u'cachelvname': u'cachelv_gluster_thinpool_gluster_vg_{disklabel}p1', u'cachedisk': u'/dev/sda4', u'cachemetalvname': u'cache_gluster_thinpool_gluster_vg_{disklabel}p1', u'cachemode': u'writeback', u'cachemetalvsize': u'70G', u'cachelvsize': u'630G'}) => {"ansible_loop_var": "item", "changed": false, "err": " Physical volume \"/dev/mapper/vdo_{disklabel}p1\" still in use\n", "item": {"cachedisk": "/dev/sda4", "cachelvname": "cachelv_gluster_thinpool_gluster_vg_{disklabel}p1", "cachelvsize": "630G", "cachemetalvname": "cache_gluster_thinpool_gluster_vg_{disklabel}p1", "cachemetalvsize": "70G", "cachemode": "writeback", "cachethinpoolname": "gluster_thinpool_gluster_vg_{disklabel}p1", "vgname": "gluster_vg_{disklabel}p1"}, "msg": "Unable to reduce gluster_vg_{disklabel}p1 by /dev/dm-15.", "rc": 5}
somewhere within that I see something that points to a race condition ("still in use").
Unfortunately I have not been able to pinpoint the raw logs which are used at that stage and I wasn't able to obtain more info.
At this point quite a bit of storage setup is already done, so rolling back for a clean new attempt, can be a bit complicated, with reboots to reconcile the kernel with data on disk.
I don't actually believe it's related to single node and I'd be quite happy to move the creation of the SSD cache to a later stage, but in a VDO setup, this looks slightly complex to someone without intimate knowledge of LVS-with-cache-and-perhaps-thin/VDO/Gluster all thrown into one.
Needless the feature set (SSD caching & compressed-dedup) sounds terribly attractive but when things don't just work, it's more terrifying.
4 years, 4 months
Moving HostedEngine
by Joseph Goldman
Hi List,
In one of my installs, I set up the first storage domain (and where
the HostedEngine is) on a bigger NFS NAS - since then I have created a
Gluster volume that spans the 3 hosts and I'm putting a few VM's in
there for higher reliability (as SAN is single point of failure) namely
I'd like to put HostedEngine in there so it stays up no matter what and
can help report if issues occur (network issue to NAS, NAS dies etc etc)
Looking through other posts and documentation, there's no real way to
move the HostedEngine storage, is this correct? The solution I've seen
is to backup the hosted engine DB, blow it away, and re-deploy it from
the .backup file configuring it to the new storage domain in the deploy
script - is this the only process? How likely is this to fail? Is it
likely that all VM's and settings will be picked straight back up and
continue to operate like normal? I dont have a test setup to play around
with atm so just trying to gauge confidence in such a solution.
Thanks,
Joe
4 years, 4 months
Migrate VM from oVirt to oVirt
by adrianquintero＠gmail.com
Hello,
What would be the procedure to migrate a VM from oVirt to oVirt?
Migrate from oVirt 4.2 running on Site A to oVirt 4.3 Site B.
thanks!
4 years, 4 months
Re: HostedEngine Deployment fails on AMD EPYC 7402P 4.3.7
by Strahil
The HostedEngine does it automatically, but the option that defines the OVF refresh interval is not accurate.
Just power it up without that option (cluster in maintenance) and keep it running for a day.
On the next day power it off and try to power it up via hosted-engine --vm-start.
Best Regards,
Strahil NikolovOn Nov 28, 2019 11:54, Ralf Schenk <rs(a)databay.de> wrote:
>
> Hello,
>
> I did something like that via "virsh edit HostedEngine".
>
> But how is the change written back to the shared storage ("hosted_storage" so it stays permanent for HA Engine ?
>
> I was able to boot up HostedEngine manually via virsh start after removing the required flag from XML (I first added a user to sasldb in /etc/libvirt/passwd.db to be able to log into libvirt).
>
> Bye
>
>
> Am 28.11.2019 um 05:51 schrieb Strahil:
>
> Hi Ralf,
> When the deployment fail - you can dump the xml from virsh , edit it, undefine the current HostedEngine and define your modified HostedEngine 's xml.
>
> Once you do that, you can try to start the
> VM.
>
> Good luck.
>
> Best Regards,
> Strahil Nikolov
>
> On Nov 27, 2019 18:28, Ralf Schenk <rs(a)databay.de> wrote:
>>
>> Hello,
>>
>> This week I tried to deploy Hosted Engine on Ovirt-Node-NG 4.3.7 based Host.
>>
>> At the time the locally deployed Engine ist copied to hosted-storage (in my case NFS) and deployment tries to start the Engine (via ovirt-ha-agent) this fails.
>>
>> QUEMU Log (/var/log/libvirt/qemu/HostedEngine.log) only shows "2019-11-27 16:17:16.833+0000: shutting down, reason=failed".
>>
>> Researching the cause is: The built Libvirt VM XML includes the feature "virt-ssbd" as requirement, which is simly not there.
>>
>> From VM XML:
>>
>> <cpu mode='custom' match='exact' check='partial'>
>> <model fallback='allow'>EPYC</model>
>> <topology sockets='16' cores='4' threads='1'/>
>> <feature policy='require' name='ibpb'/>
>> <feature policy='require' name='virt-ssbd'/>
>>
>> from cat /proc/cpuinfo:
>>
>> processor : 47
>> vendor_id : AuthenticAMD
>> cpu family : 23
>> model : 49
>> model name : AMD EPYC 7402P 24-Core Processor
>> stepping : 0
>> microcode : 0x830101c
>> cpu MHz : 2800.000
>> cache size : 512 KB
>> physical id : 0
>> siblings : 48
>> core id : 30
>> cpu cores : 24
>> api
4 years, 4 months
Re: spice connection error
by Strahil
Check on host 'node3' in the Network configuration if you have a network out of sync or any clues about the error .
Best Regards,
Strahil NikolovOn Nov 28, 2019 11:44, Kim Kargaard <Kim.Kargaard(a)noroff.no> wrote:
>
> Hi,
>
> I am getting the attached error when trying to move the Display from the one network to the other within the cluster. I can't see any place to set an IP for the vnic that I want to set as the display network.
>
> Any thoughts?
>
> Kim
>
> On 28/11/2019, 06:41, "Strahil" <hunter86_bg(a)yahoo.com> wrote:
>
> As far as I know , the engine plays a role as a proxy during the establishment of the connection.
> Check that you can reach both engine and the host from your system.
>
> For the same reason, I use noVNC - as you just need a single port to the engine in addition to the rest of the settings.
>
> Best Regards,
> Strahil NikolovOn Nov 27, 2019 11:27, kim.kargaard(a)noroff.no wrote:
> >
> > Hi,
> >
> > When trying to connect from a remote network on the spice console to a VM, I get the following error:
> >
> > (remote-viewer:80195): virt-viewer-WARNING **: 11:05:22.322: Channel error: Could not connect to proxy server xx.xx.xx.xx: Socket I/O timed out
> >
> > I found that the display is set to the management network and not the VM networkn in the cluster logical network. However, when I try to set the other vlan to be the display network, I get the following error:
> >
> > Error while executing action: Cannot edit Network. IP address has to be set for the NIC that bears a role network. Network: student-vlan100, Nic: p2p1.100 on host node3 violates that rule.
> >
> > I am not sure what this means. Any ideas?
> >
> > Kind regards
> >
> > Kim
> > _______________________________________________
> > Users mailing list -- users(a)ovirt.org
> > To unsubscribe send an email to users-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6YEI3I4NCAH...
>
>
4 years, 4 months
VDSM Errors see below
by rob.downer＠orbitalsystems.co.uk
I have removed all deployment of the hosted engine by running the following commands.
ovirt-hosted-engine-cleanup
vdsm-tool configure --force
systemctl restart libvirtd
systemctl restart vdsm
on my hosts I have the following ovirt 1 is the host I ran the hosted engine setup.
I have set the Gluster network to use the same subnet and set up forward and reverse DNS for the Gluster port network NIC's
I had this working using a separate subnet but thought to try it on the same subnet to avoid any issues that may have occurred while using a separate network subnet.
the main Host IP address is still showing in Unamanaged Connections on Ovirt 1 .. is this anything to be concerned about after running the commands above...
I have restarted all machines.
All come back with these VDSM errors...
Node 1
[root@ovirt1 ~]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2019-11-28 13:29:40 UTC; 37min ago
Process: 31178 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)
Process: 30721 ExecStart=/usr/share/vdsm/daemonAdapter -0 /dev/null -1 /dev/null -2 /dev/null /usr/share/vdsm/vdsmd (code=exited, status=0/SUCCESS)
Main PID: 30721 (code=exited, status=0/SUCCESS)
Nov 26 22:42:49 ovirt1.kvm.private vdsm[30721]: WARN MOM not available, KSM stats will be missing.
Nov 26 22:42:49 ovirt1.kvm.private vdsm[30721]: WARN Not ready yet, ignoring event '|virt|VM_status|871ce9d5-417a-4278-8446-28b681760c1b' args={'871ce9d5-417a-4278-8446-28b681760c1b': {'status': 'Poweri...
Nov 28 13:28:43 ovirt1.kvm.private vdsm[30721]: WARN File: /var/run/vdsm/trackedInterfaces/eno2 already removed
Nov 28 13:29:26 ovirt1.kvm.private vdsm[30721]: WARN File: /var/lib/libvirt/qemu/channels/871ce9d5-417a-4278-8446-28b681760c1b.com.redhat.rhevm.vdsm already removed
Nov 28 13:29:26 ovirt1.kvm.private vdsm[30721]: WARN File: /var/lib/libvirt/qemu/channel/target/domain-1-HostedEngineLocal/org.qemu.guest_agent.0 already removed
Nov 28 13:29:39 ovirt1.kvm.private vdsm[30721]: WARN MOM not available.
Nov 28 13:29:39 ovirt1.kvm.private vdsm[30721]: WARN MOM not available, KSM stats will be missing.
Nov 28 13:29:39 ovirt1.kvm.private systemd[1]: Stopping Virtual Desktop Server Manager...
Nov 28 13:29:39 ovirt1.kvm.private vdsmd_init_common.sh[31178]: vdsm: Running run_final_hooks
Nov 28 13:29:40 ovirt1.kvm.private systemd[1]: Stopped Virtual Desktop Server Manager.
Hint: Some lines were ellipsized, use -l to show in full.
[root@ovirt1 ~]# nodectl check
Status: WARN
Bootloader ... OK
Layer boot entries ... OK
Valid boot entries ... OK
Mount points ... OK
Separate /var ... OK
Discard is used ... OK
Basic storage ... OK
Initialized VG ... OK
Initialized Thin Pool ... OK
Initialized LVs ... OK
Thin storage ... OK
Checking available space in thinpool ... OK
Checking thinpool auto-extend ... OK
vdsmd ... BAD
NODE 2
[root@ovirt2 ~]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-11-28 13:57:30 UTC; 1min 13s ago
Process: 3626 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
Main PID: 5418 (vdsmd)
Tasks: 38
CGroup: /system.slice/vdsmd.service
└─5418 /usr/bin/python2 /usr/share/vdsm/vdsmd
Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: libvirt: Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing'
Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running dummybr
Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running tune_system
Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running test_space
Nov 28 13:57:30 ovirt2.kvm.private vdsmd_init_common.sh[3626]: vdsm: Running test_lo
Nov 28 13:57:30 ovirt2.kvm.private systemd[1]: Started Virtual Desktop Server Manager.
Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN File: /var/run/vdsm/trackedInterfaces/eno1 already removed
Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN File: /var/run/vdsm/trackedInterfaces/eno2 already removed
Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN MOM not available.
Nov 28 13:57:32 ovirt2.kvm.private vdsm[5418]: WARN MOM not available, KSM stats will be missing.
[root@ovirt2 ~]# nodectl check
Status: OK
Bootloader ... OK
Layer boot entries ... OK
Valid boot entries ... OK
Mount points ... OK
Separate /var ... OK
Discard is used ... OK
Basic storage ... OK
Initialized VG ... OK
Initialized Thin Pool ... OK
Initialized LVs ... OK
Thin storage ... OK
Checking available space in thinpool ... OK
Checking thinpool auto-extend ... OK
vdsmd ... OK
[root@ovirt2 ~]#
Node 3
[root@ovirt3 ~]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-11-28 13:57:08 UTC; 12min ago
Process: 2849 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
Main PID: 3220 (vdsmd)
Tasks: 38
CGroup: /system.slice/vdsmd.service
└─3220 /usr/bin/python2 /usr/share/vdsm/vdsmd
Nov 28 13:57:07 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running nwfilter
Nov 28 13:57:07 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running dummybr
Nov 28 13:57:08 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running tune_system
Nov 28 13:57:08 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running test_space
Nov 28 13:57:08 ovirt3.kvm.private vdsmd_init_common.sh[2849]: vdsm: Running test_lo
Nov 28 13:57:08 ovirt3.kvm.private systemd[1]: Started Virtual Desktop Server Manager.
Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN File: /var/run/vdsm/trackedInterfaces/eno1 already removed
Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN File: /var/run/vdsm/trackedInterfaces/eno2 already removed
Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN MOM not available.
Nov 28 13:57:09 ovirt3.kvm.private vdsm[3220]: WARN MOM not available, KSM stats will be missing.
[root@ovirt3 ~]# nodectl check
Status: OK
Bootloader ... OK
Layer boot entries ... OK
Valid boot entries ... OK
Mount points ... OK
Separate /var ... OK
Discard is used ... OK
Basic storage ... OK
Initialized VG ... OK
Initialized Thin Pool ... OK
Initialized LVs ... OK
Thin storage ... OK
Checking available space in thinpool ... OK
Checking thinpool auto-extend ... OK
vdsmd ... OK
[root@ovirt3 ~]#
4 years, 4 months
