May 2019 - Users - oVirt List Archives

Active-Passive Disaster Recovery
by Paulo Silva 26 May '19

26 May '19

Hi, I want to setup an Active-Passive between 2 sites using oVirt 4.3. For now I have a primary site with an ovirt-engine, 2 ovirt hosts and 5 VMs in an iscsi storage. The backup site will have an ovirt-engine with 2 ovirt hosts and an iscsi storage that will will have the same LUNs of the primary replicated. I'm reading the DR Guide ( https://ovirt.org/documentation/disaster-recovery-guide/) in the backup site do I need to manually create the storage domains and the VM networks or will the ansible playbook take care of that? Thanks -- Paulo Silva <paulojjs(a)gmail.com>

3 2

Single instance scaleup.
by Leo David 26 May '19

26 May '19

Hello Everyone, Can someone help me to clarify this ? I have a single-node 4.2.8 installation ( only two gluster storage domains - distributed single drive volumes ). Now I just got two identintical servers and I would like to go for a 3 nodes bundle. Is it possible ( after joining the new nodes to the cluster ) to expand the existing volumes across the new nodes and change them to replica 3 arbitrated ? If so, could you share with me what would it be the procedure ? Thank you very much ! Leo

1 0

Unable to find OVF_STORE after recovery / upgrade
by Sam Cappello 25 May '19

25 May '19

This is a multi-part message in MIME format. --------------05F3D0780062C6D37C1619A7 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, so i was running a 3.4 hosted engine two node setup on centos 6, had some disk issues so i tried to upgrade to centos 7 and follow the path 3.4 > 3.5 > 3.6 > 4.0. i screwed up dig time somewhere between 3.6 and 4.0, so i wiped the drives, installed a fresh 4.0.3, then created the database and restored the 3.6 engine backup before running engine-setup as per the docs. things seemed to work, but i have the the following issues / symptoms: - ovirt-ha-agent running 100% CPU on both nodes - messages in the UI that the Hosted Engine storage Domain isn't active and Failed to import the Hosted Engine Storage Domain - hosted engine is not visible in the UI and the following repeating in the agent.log: MainThread::INFO::2016-10-03 12:38:27,718::hosted_engine::461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) MainThread::INFO::2016-10-03 12:38:27,720::hosted_engine::466::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host vmhost1.oracool.net (id: 1, score: 3400) MainThread::INFO::2016-10-03 12:38:37,979::states::421::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2016-10-03 12:38:37,985::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-03 12:38:45,645::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-03 12:38:45,647::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-03 12:39:00,543::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-03 12:39:00,562::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-03 12:39:01,235::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-03 12:39:01,236::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2016-10-03 12:39:09,295::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain MainThread::INFO::2016-10-03 12:39:09,296::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-10-03 12:39:16,928::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-10-03 12:39:16,934::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf I have searched a bit and not really found a solution, and have come to the conclusion that i have made a mess of things, and am wondering if the best solution is to export the VMs, and reinstall everything then import them back? i am using remote NFS storage. if i try and add the hosted engine storage domain it says it is already registered. i have also upgraded and am now running oVirt Engine Version: 4.0.4.4-1.el7.centos hosts were installed using ovirt-node. currently at 3.10.0-327.28.3.el7.x86_64 if a fresh install is best, any advice / pointer to doc that explains best way to do this? i have not moved my most important server over to this cluster yet so i can take some downtime to reinstall. thanks! sam --------------05F3D0780062C6D37C1619A7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi, so i was running a 3.4 hosted engine two node setup on centos 6, had some disk issues so i tried to upgrade to centos 7 and follow the path 3.4 > 3.5 > 3.6 > 4.0. i screwed up dig time somewhere between 3.6 and 4.0, so i wiped the drives, installed a fresh 4.0.3, then created the database and restored the 3.6 engine backup before running engine-setup as per the docs. things seemed to work, but i have the the following issues / symptoms: - ovirt-ha-agent running 100% CPU on both nodes - messages in the UI that the Hosted Engine storage Domain isn't active and Failed to import the Hosted Engine Storage Domain - hosted engine is not visible in the UI and the following repeating in the agent.log: MainThread::INFO::2016-10-03 12:38:27,718::hosted_engine::461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 3400) MainThread::INFO::2016-10-03 12:38:27,720::hosted_engine::466::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host vmhost1.oracool.net (id: 1, score: 3400) MainThread::INFO::2016-10-03 12:38:37,979::states::421::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2016-10-03 12:38:37,985::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-03 12:38:45,645::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-03 12:38:45,647::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-03 12:39:00,543::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-03 12:39:00,562::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-03 12:39:01,235::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-03 12:39:01,236::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2016-10-03 12:39:09,295::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Reloading vm.conf from the shared storage domain MainThread::INFO::2016-10-03 12:39:09,296::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-10-03 12:39:16,928::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-10-03 12:39:16,934::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf I have searched a bit and not really found a solution, and have come to the conclusion that i have made a mess of things, and am wondering if the best solution is to export the VMs, and reinstall everything then import them back? i am using remote NFS storage. if i try and add the hosted engine storage domain it says it is already registered. i have also upgraded and am now running oVirt Engine Version: 4.0.4.4-1.el7.centos hosts were installed using ovirt-node. currently at 3.10.0-327.28.3.el7.x86_64 if a fresh install is best, any advice / pointer to doc that explains best way to do this? i have not moved my most important server over to this cluster yet so i can take some downtime to reinstall. thanks! sam </body> </html> --------------05F3D0780062C6D37C1619A7--

3 2

[Feedback Needed] oVirt metrics store
by Shirly Radco 23 May '19

23 May '19

Hi, In the latest oVirt 4.3.z and 4.2.z releases we have simplified the metrics store installation immensely and updated it to be based on OpenShift 3.11, with Elasticsearch and Kibana 5. The oVirt metrics store provides monitoring for the oVirt hosts and vms, for both metrics and logs with pre-built example dashboards and self service visualizations in Kibana. 5 steps for installing oVirt metrics store: 1. Update variables files 2. Run 1st ansible playbook -> creates the vms 3. Run 2nd ansible playbook -> deploys the metrics store on the created vms 4. Run 3rd ansible playbook -> configures the data collectors on the oVirt hosts and engine 4. Load example dashboards to Kibana In addition, we have replaced Fluentd with Rsyslog in 4.3.z hosts, for better performance. I invite you to try it out and give us feedback. - Installation guide - https://ovirt.org/documentation/metrics-install-guide/metrics_store_install… - User guide - https://ovirt.org/documentation/metrics-user-guide/metrics-user-guide.html - Short installation video <https://dms.licdn.com/playback/C5605AQHirxO3r-3OFw/14a815a1904945ac83774d7e…> Best regards, -- SHIRLY RADCO BI SENIOR SOFTWARE ENGINEER Red Hat Israel <https://www.redhat.com/> <https://red.ht/sig>

1 0

enabling vPMU for virtual machine
by Jarosław Prokopowski 23 May '19

23 May '19

Hi Guys, Is there a way to enable vPMU for VM in oVirt? I already enabled Nested Virtualization on hosts and installed vdsm-hook-nestedvt. In libvirt it is possible to set <cpu mode='host-passthrough'/> This blog post shows how to set it in libvirt xml vm config: https://stackoverflow.com/questions/46490857/how-to-enable-pmu-in-kvm-guest A clue from Red Hat: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… Maybe something with cpuflags? https://mpolednik.github.io/2017/07/31/cpuflags-ovirt-hook-cpu-features/ Thanks Jarek

1 1

"Upgrade" from oVirt to RHV
by Vinícius Ferrão 23 May '19

23 May '19

Hello, I would like to know if there’s a supported path to move from oVirt to RHV. oVirt is running on version 4.3.0.4-1.el7. RHV would be version 4.3.3.7-0.1.el7. I was thinking in reinstalling a host with RHV 4.3 and adding it to the oVirt HE. Move all the VM’s to the RHV host and than do the same process. After it install a new RHVH-M to remove the oVirt Engine. Not sure if this would work or not. Thanks,

4 3

Unable to deploy Hyperconverged Engine Node - v4.3.3
by anonmix＠gmail.com 22 May '19

22 May '19

Hi everyone, I am trying a Gluster Hyperconvergence deployment where the Gluster part has been completed successfully. All hosts are Centos 7.6.1810 (fresh install) and two HP DL20 G9 (for VM's) and one HP 120 G7 (which hosts the Gluster arbiter volumes). Unfortunately I am unable to deploy the Engine, both CLI and GUI approaches fail with the error below. On first sight it looks similar to https://lists.ovirt.org/pipermail/users/2018-March/087802.html but I've configured a static IP (same subnet as the host), no DHCP. I also tried to force ipv4 with "/usr/sbin/ovirt-hosted-engine-setup --4" but the very same error was thrown in every case when trying to deploy the engine: [ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": [{"address": "sub.sub.domain.tld", "affinity_labels": [], "auto_numa_status": "unknown", "certificate": {"organization": "sub.domain.tld", "subject": "O=sub.domain.tld,CN=sub.sub.domain.tld"}, "cluster": {"href": "/ovirt-engine/api/clusters/f083f056-74fd-11e9-bba9-00163e522076", "id": "f083f056-74fd-11e9-bba9-00163e522076"}, "comment": "", "cpu": {"speed": 0.0, "topology": {}}, "device_passthrough": {"enabled": false}, "devices": [], "external_network_provider_configurations": [], "external_status": "ok", "hardware_information": {"supported_rng_sources": []}, "hooks": [], "href": "/ovirt-engine/api/hosts/dc4f5c15-4989-4454-ba46-3bd600796b69", "id": "dc4f5c15-4989-4454-ba46-3bd600796b69", "katello_errata": [], "kdump_status": "unknown", "ksm": {"enabled": false}, "max_scheduling_memory": 0, "memory": 0, "name": "sub.sub.domain.tld", "network_attachments": [], "nics": [], "numa_nodes": [], "numa_supported": fals e, "os": {"custom_kernel_cmdline": ""}, "permissions": [], "port": 54321, "power_management": {"automatic_pm_enabled": true, "enabled": false, "kdump_detection": true, "pm_proxies": []}, "protocol": "stomp", "se_linux": {}, "spm": {"priority": 5, "status": "none"}, "ssh": {"fingerprint": "SHA256:L8YyAMcxLFJEng+CoDympwkpMwoagcBafI4fpLP4Kk0", "port": 22}, "statistics": [], "status": "install_failed", "storage_connection_extensions": [], "summary": {"total": 0}, "tags": [], "transparent_huge_pages": {"enabled": false}, "type": "rhel", "unmanaged_networks": [], "update_available": false, "vgpu_placement": "consolidated"}]}, "attempts": 120, "changed": false} Unfortunately I don't really have an idea where to check for what considering the error message. The to be deployed engine VM gets listed as KVM VM, is accessible through the bridge and seems to be started up completely, I can even access the Engine web interface (engine01.sub.domain.tld/ovirt-engine). In /var/log/messages the following can be found ... "May 13 12:40:55 host ansible-async_wrapper.py: 15505 still running (86015) May 13 12:40:57 host python: ansible-ovirt_host_facts Invoked with all_content=False pattern=name=sub.sub.domain.tld fetch_nested=False nested_attributes=[] auth={'timeout': 0, 'url': 'https://engine01.sub.domain.tld/ovirt-engine/api', 'insecure': True, 'kerberos': False, 'compress': True, 'headers': None, 'token': '8s-vELzQqNTR6l7-KRuqnYLE3sVwVWU5NxiNWzc-s2CllaQG_5YZ32fCFkVsAgwEyLWjPIOxvyS-_4js-VYFFQ', 'ca_file': None}" ... and after 120 attempts Ansible stops and fails with a deployment error. When re-trying after removing the VM and ovirt-hosted-engine-cleanup the very same error is thrown. What is a bit weird is this entry in /var/log/ovirt-hosted-engine-setup/ ./engine-logs-2019-05-13T12:26:20Z/ovirt-engine/engine.log:2019-05-13 12:34:40,369Z ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [12746235] SSH error running command root(a)sub.sub.domain.tld:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': RuntimeException: Unexpected error during execution: bash: /tmp/ovirt-pTVEEzlb8b/ovirt-host-deploy: Permission denied ./engine-logs-2019-05-13T12:26:20Z/ovirt-engine/engine.log:2019-05-13 12:34:40,406Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [12746235] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host sub.sub.domain.tld: Unexpected error during execution: bash: /tmp/ovirt-pTVEEzlb8b/ovirt-host-deploy: Permission denied Could that be the cause and how can I fix it? What else do you guys need? Thanks in advance, Martin

4 6

Engine restore errors out on "Wait for OVF_STORE disk content"
by Andreas Elvers 22 May '19

22 May '19

Hello, when trying to deploy the engine using "hosted-engine --deploy --restore-from-file=myenginebackup" the ansible playbook errors out at [ INFO ] TASK [ovirt.hosted_engine_setup : Trigger hosted engine OVF update and enable the serial console] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Wait until OVF update finishes] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Parse OVF_STORE disk list] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Check OVF_STORE volume status] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE disk content] [ ERROR ] {u'_ansible_parsed': True, u'stderr_lines': [u'20+0 records in', u'20+0 records out', u'10240 bytes (10 kB) copied, 0.000141645 s, 72.3 MB/s', u'tar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in archive', u'tar: Exiting with failure status due to previous errors'], u'changed': True, u'end': u'2019-05-08 15:21:47.595195', u'_ansible_item_label': {u'image_id': u'65fd6c57-033c-4c95-87c1-b16c26e4bc98', u'name': u'OVF_STORE', u'id': u'9ff8b389-5e24-4166-9842-f1d6104b662b'}, u'stdout': u'', u'failed': True, u'_ansible_item_result': True, u'msg': u'non-zero return code', u'rc': 2, u'start': u'2019-05-08 15:21:46.906877', u'attempts': 12, u'cmd': u"vdsm-client Image prepare storagepoolID=597f329c-0296-03af-0369-000000000139 storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8 imageID=9ff8b389-5e24-4166-9842-f1d6104b662b volumeID=65fd6c57-033c-4c95-87c1-b16c26e4bc98 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - ebb09b0e-2d03-40f0-8fa4-c40 b18612a54.ovf", u'item': {u'image_id': u'65fd6c57-033c-4c95-87c1-b16c26e4bc98', u'name': u'OVF_STORE', u'id': u'9ff8b389-5e24-4166-9842-f1d6104b662b'}, u'delta': u'0:00:00.688318', u'invocation': {u'module_args': {u'warn': False, u'executable': None, u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare storagepoolID=597f329c-0296-03af-0369-000000000139 storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8 imageID=9ff8b389-5e24-4166-9842-f1d6104b662b volumeID=65fd6c57-033c-4c95-87c1-b16c26e4bc98 | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr': u'20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied, 0.000141645 s, 72.3 MB/s\ntar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors', u'_ansible_no_log': False} [ ERROR ] {u'_ansible_parsed': True, u'stderr_lines': [u'20+0 records in', u'20+0 records out', u'10240 bytes (10 kB) copied, 0.000140541 s, 72.9 MB/s', u'tar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in archive', u'tar: Exiting with failure status due to previous errors'], u'changed': True, u'end': u'2019-05-08 15:24:01.387469', u'_ansible_item_label': {u'image_id': u'dacf9ad8-77b9-4205-8ca2-d6877627ad4a', u'name': u'OVF_STORE', u'id': u'8691076a-8e45-4429-a18a-5faebef866cc'}, u'stdout': u'', u'failed': True, u'_ansible_item_result': True, u'msg': u'non-zero return code', u'rc': 2, u'start': u'2019-05-08 15:24:00.660309', u'attempts': 12, u'cmd': u"vdsm-client Image prepare storagepoolID=597f329c-0296-03af-0369-000000000139 storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8 imageID=8691076a-8e45-4429-a18a-5faebef866cc volumeID=dacf9ad8-77b9-4205-8ca2-d6877627ad4a | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - ebb09b0e-2d03-40f0-8fa4-c40 b18612a54.ovf", u'item': {u'image_id': u'dacf9ad8-77b9-4205-8ca2-d6877627ad4a', u'name': u'OVF_STORE', u'id': u'8691076a-8e45-4429-a18a-5faebef866cc'}, u'delta': u'0:00:00.727160', u'invocation': {u'module_args': {u'warn': False, u'executable': None, u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare storagepoolID=597f329c-0296-03af-0369-000000000139 storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8 imageID=8691076a-8e45-4429-a18a-5faebef866cc volumeID=dacf9ad8-77b9-4205-8ca2-d6877627ad4a | grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf - ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'stderr': u'20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied, 0.000140541 s, 72.9 MB/s\ntar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in archive\ntar: Exiting with failure status due to previous errors', u'_ansible_no_log': False} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook I tried twice. Same result. Should I retry? Is it safe to use the local hosted engine for starting stopping vms? I'm kind of headless for some days :-) Best regards.

3 11

ovirt 4.3.3.7 cannot create a gluster storage domain
by Strahil Nikolov 22 May '19

22 May '19

Hey guys, I have recently updated (yesterday) my platform to latest available (v4.3.3.7) and upgraded to gluster v6.1 .The setup is hyperconverged 3 node cluster with ovirt1/gluster1 & ovirt2/gluster2 as replica nodes (glusterX is for gluster communication) while ovirt3 is the arbiter. Today I have tried to add new domain storages but they fail with the following: 2019-05-16 10:15:21,296+0300 INFO (jsonrpc/2) [vdsm.api] FINISH createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0 kB/s\n" from=::ffff:192.168.1.2,43864, flow_id=4a54578a, task_id=d2535d0f-c7f7-4f31-a10f-704923ce1790 (api:52) 2019-05-16 10:15:21,296+0300 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='d2535d0f-c7f7-4f31-a10f-704923ce1790') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in createStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2614, in createStorageDomain storageType, domVersion, block_size, alignment) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 106, in create block_size) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 466, in _prepareMetadata cls.format_external_leases(sdUUID, xleases_path) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 1255, in format_external_leases xlease.format_index(lockspace, backend) File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 681, in format_index index.dump(file) File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 843, in dump file.pwrite(INDEX_BASE, self._buf) File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1076, in pwrite self._run(args, data=buf[:]) File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1093, in _run raise cmdutils.Error(args, rc, "[suppressed]", err) Error: Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0 kB/s\n" 2019-05-16 10:15:21,296+0300 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='d2535d0f-c7f7-4f31-a10f-704923ce1790') aborting: Task is aborted: u'Command [\'/usr/bin/dd\', \'iflag=fullblock\', u\'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases\', \'oflag=direct,seek_bytes\', \'seek=1048576\', \'bs=256512\', \'count=1\', \'conv=notrunc,nocreat,fsync\'] failed with rc=1 out=\'[suppressed]\' err="/usr/bin/dd: error writing \'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases\': Invalid argument\\n1+0 records in\\n0+0 records out\\n0 bytes (0 B) copied, 0.0138582 s, 0.0 kB/s\\n"' - code 100 (task:1181) 2019-05-16 10:15:21,297+0300 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0 kB/s\n" (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error Error: Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/glusterSD/gluster1:_data__fast2/591d9b61-5c7d-4388-a6b7-ab03181dff8a/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0138582 s, 0.0 kB/s\n" 2019-05-16 10:15:21,297+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 351) in 0.45 seconds (__init__:312) 2019-05-16 10:15:22,068+0300 INFO (jsonrpc/1) [vdsm.api] START disconnectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'mnt_options': u'backup-volfile-servers=gluster2:ovirt3', u'id': u'7442e9ab-dc54-4b9a-95d9-5d98a1e81b05', u'connection': u'gluster1:/data_fast2', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.1.2,43864, flow_id=33ced9b2-cdd5-4147-a223-d0eb398a2daf, task_id=a9a8f90a-1603-40c6-a959-3cbff29d1d7b (api:48) 2019-05-16 10:15:22,068+0300 INFO (jsonrpc/1) [storage.Mount] unmounting /rhev/data-center/mnt/glusterSD/gluster1:_data__fast2 (mount:212) I have tested manually mounting and trying it again: [root@ovirt1 logs]# mount -t glusterfs -o backupvolfile-server=gluster2:ovirt3 gluster1:/data_fast2 /mnt [root@ovirt1 logs]# cd /mnt/ [root@ovirt1 mnt]# ll total 0 [root@ovirt1 mnt]# dd if=/dev/zero of=file bs=4M status=progress count=250 939524096 bytes (940 MB) copied, 8.145447 s, 115 MB/s 250+0 records in 250+0 records out 1048576000 bytes (1.0 GB) copied, 9.08347 s, 115 MB/s [root@ovirt1 mnt]# /usr/bin/dd iflag=fullblock of=file oflag=direct,seek_bytes seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync status=progress ^C0+0 records in 0+0 records out 0 bytes (0 B) copied, 46.5877 s, 0.0 kB/s Can someone give a hint ? Maybe it's related to gluster v6 ? Can someone test with older version of Gluster ? Best Regards,Strahil Nikolov

3 8

Error while executing action Change CD: Drive image file could not be found
by racevedo＠lenovo.com 22 May '19

22 May '19

I see that this was apparently a bug in previous version, but I'm currently using oVirt Node 4.3.3.1. This was working fine before updating from oVirt Node 4.2. This occurs when trying to change the CD on a running VM. Log shows: 2019-05-03 12:09:24,731-04 ERROR [org.ovirt.engine.core.bll.storage.disk.ChangeDiskCommand] (default task-52) [309a3a99-c754-45d5-add7-c45e094d2e69] Command 'org.ovirt.engine.core.bll.storage.disk.ChangeDiskCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ChangeDiskVDS, error = Drive image file could not be found, code = 13 (Failed with error imageErr and code 13) Any advice on how to solve this Current configuration: OS Version : RHEL - 7 - 6.1810.2.el7.centos OS Description : oVirt Node 4.3.3.1 Kernel Version : 3.10.0 - 957.10.1.el7.x86_64 KVM Version : 2.12.0 - 18.el7_6.3.1 LIBVIRT Version : libvirt-4.5.0-10.el7_6.6 VDSM Version : vdsm-4.30.13-1.el7

2 2

[ANN] oVirt 4.3.4 Second Release Candidate is now available
by Sandro Bonazzola 22 May '19

22 May '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.4 Second Release Candidate, as of May 22nd, 2019. This update is a release candidate of the fourth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used inproduction. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available[2] Additional Resources: * Read more about the oVirt 4.3.4 release highlights: http://www.ovirt.org/release/4.3.4/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.4/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>

1 0

vGPU becomes very inefficient when I use it in ovirt instead of qemu-kvm
by weizhengya16＠gmail.com 22 May '19

22 May '19

We distribute vGpu in order to use it in virtual machines. I find the same type vGpu get differently performance in Ovirt and Qemu-kvm . when I distribute a vGPU to Ovirt virtual machine,it's performance becomes very strange.Some games can run efficiently and attach 200+ fps.However,in some other games,it's can't even attach 10 FPS. But in Qemu-kvm,the vGpu's performance always efficiently,can run these games all in 200+fps. I've been stuck with this for weeks and I have to make it always runs efficiently. Can someone help me?Thank you very very very much....

2 2

Re: [ovirt-announce] Re: [ANN] oVirt 4.3.4 First Release Candidate is now available
by Strahil 22 May '19

22 May '19

Dear Krutika, Yes I did but I use 6 ports (1 gbit/s each) and this is the reason that reads get slower. Do you know a way to force gluster to open more connections (client to server & server to server)? Thanks for the detailed explanation. Best Regards, Strahil NikolovOn May 21, 2019 08:36, Krutika Dhananjay <kdhananj(a)redhat.com> wrote: > > So in our internal tests (with nvme ssd drives, 10g n/w), we found read performance to be better with choose-local > disabled in hyperconverged setup. See https://bugzilla.redhat.com/show_bug.cgi?id=1566386 for more information. > > With choose-local off, the read replica is chosen randomly (based on hash value of the gfid of that shard). > And when it is enabled, the reads always go to the local replica. > We attributed better performance with the option disabled to bottlenecks in gluster's rpc/socket layer. Imagine all read > requests lined up to be sent over the same mount-to-brick connection as opposed to (nearly) randomly getting distributed > over three (because replica count = 3) such connections. > > Did you run any tests that indicate "choose-local=on" is giving better read perf as opposed to when it's disabled? > > -Krutika > > On Sun, May 19, 2019 at 5:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Ok, >> >> so it seems that Darell's case and mine are different as I use vdo. >> >> Now I have destroyed Storage Domains, gluster volumes and vdo and recreated again (4 gluster volumes on a single vdo). >> This time vdo has '--emulate512=true' and no issues have been observed. >> >> Gluster volume options before 'Optimize for virt': >> >> Volume Name: data_fast >> Type: Replicate >> Volume ID: 378804bf-2975-44d8-84c2-b541aa87f9ef >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gluster1:/gluster_bricks/data_fast/data_fast >> Brick2: gluster2:/gluster_bricks/data_fast/data_fast >> Brick3: ovirt3:/gluster_bricks/data_fast/data_fast (arbiter) >> Options Reconfigured: >> transport.address-family: inet >> nfs.disable: on >> performance.client-io-threads: off >> cluster.enable-shared-storage: enable >> >> Gluster volume after 'Optimize for virt': >> >> Volume Name: data_fast >> Type: Replicate >> Volume ID: 378804bf-2975-44d8-84c2-b541aa87f9ef >> Status: Stopped >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gluster1:/gluster_bricks/data_fast/data_fast >> Brick2: gluster2:/gluster_bricks/data_fast/data_fast >> Brick3: ovirt3:/gluster_bricks/data_fast/data_fast (arbiter) >> Options Reconfigured: >> network.ping-timeout: 30 >> performance.strict-o-direct: on >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> server.event-threads: 4 >> client.event-threads: 4 >> cluster.choose-local: off >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> transport.address-family: inet >> nfs.disable: on >> performance.client-io-threads: on >> cluster.enable-shared-storage: enable >> >> After that adding the volumes as storage domains (via UI) worked without any issues. >> >> Can someone clarify why we have now 'cluster.choose-local: off' when in oVirt 4.2.7 (gluster v3.12.15) we didn't have that ? >> I'm using storage that is faster than network and reading from local brick gives very high read speed. >> >> Best Regards, >> Strahil Nikolov >> >> >> >> В неделя, 19 май 2019 г., 9:47:27 ч. Г�

2 1

Supporting comments on ovirt-site Blog section
by Roy Golan 22 May '19

22 May '19

It would be very useful to have a comment section on our ovirt site. It is quite standard to have that on every blog out there, and for a reason - you get feedback and conversation around the topic, without going somewhere else (users list, irc, etc...) What do we need to do to help the middle man with that? Regards, Roy

4 3

Re: Scale out ovirt 4.3 (from 3 to 6 or 9 nodes) with hyperconverged setup and Gluster
by Strahil 21 May '19

21 May '19

I have edited my multipath.conf to exclude local disks , but you need to set '#VDSM private' as per the comments in the header of the file. Otherwise, use the /dev/mapper/multipath-device notation - as you would do with any linux. Best Regards, Strahil NikolovOn Apr 23, 2019 01:07, adrianquintero(a)gmail.com wrote: > > Thanks Alex, that makes more sense now while trying to follow the instructions provided I see that all my disks /dev/sdb, /dev/sdc, /dev/sdd are locked and inidicating " multpath_member" hence not letting me create new bricks. And on the logs I see > > Device /dev/sdb excluded by a filter.\n", "item": {"pvname": "/dev/sdb", "vgname": "gluster_vg_sdb"}, "msg": "Creating physical volume '/dev/sdb' failed", "rc": 5} > Same thing for sdc, sdd > > Should I manually edit the filters inside the OS, what will be the impact? > > thanks again. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FW3IR3NMQTYZL…

7 19

Gluster rebuild: request suggestions (poor IO performance)
by Jim Kusznir 21 May '19

21 May '19

Hi: I've been having one heck of a time for nearly the entire time I've been running ovirt with disk IO performance. I've tried a variety of things, I've posted to this list for help several times, and it sounds like in most cases the problems are due to design decisions and such. My cluster has been devolving into nearly unusable performance, and I believe its mostly disk IO related. I'm currently using FreeNAS as my primary VM storage (via NFS), but now it too is performing slowly (it started out reasonable, but slowly devolved for unknown reasons). I'm ready to switch back to gluster if I can get specific recommendations as to what I need to do to make it work. I feel like I've been trying random things, and sinking money into this to try and make it work, but nothing has really fixed the problem. I have 3 Dell R610 servers with 750GB SSDs as their primary drive. I had used some Seagate SSHDs, but the internal Dell DRAC raid controller (which had been configured to pass them through as a single disk volume, but still wasn't really JBOD), but it started silently failing them, and causing major issues for gluster. I think the DRAC just doesn't like those HDDs. I can put some real spinning disks in; perhaps a RAID-1 pair of 2TB? These servers only take 2.5" hdd's, so that greatly limits my options. I'm sure others out there are using Dell R610 servers...what do you use for storage? How does it perform? What do I need to do to get this cluster actually usable again? Are PERC-6i storage controllers usable? I'm not even sure where to go troubleshooting now...everything is so sloooowwwww. BTW: I had a small data volume on the SSDs, and the gluster performance on those was pretty poor. performance of the hosted engine is pretty poor still, and it is still on the SSDs.

1 0

Re: Scale out ovirt 4.3 (from 3 to 6 or 9 nodes) with hyperconverged setup and Gluster
by Strahil 21 May '19

21 May '19

> EUREKA: After doing the above I was able to get past the filter issues, however I am still concerned if during a reboot the disks might come up differently. For example /dev/sdb might come up as /dev/sdx... Even if they change , you don't have to worry about as each PV contains LVM metadata (including VG configuration) which is read by LVM on boot (actually everything that is not in the LVM filter is being scanned like that). Once all PVs are available the VG is activated and then the LVs are also activated. > I am trying to make sure this setup is always the same as we want to move this to production, however seems I still don't have the full hang of it and the RHV 4.1 course is way to old :) > > Thanks again for helping out with this. It's a plain KVM with some management layer. Just a hint: Get your HostedEngine's configuration xml from the vdsm log (for emergencies) and another copy with reverse boot order where DVD is booting first. Also get the xml for the ovirtmgmt network. It helped me a lot of times when I wanted to recover my HostedEngine. I'm too lazy to rebuild it. Hint2: Vdsm logs contain each VM's configuration xml when the VMs are powered on. Hint3: Get regular backups of the HostedEngine and patch it from time to time. I would go in prod as follows: Let's say you are on 4.2.8 Next step would be to go to 4.3.latest and then to 4.4.latest . A test cluster (even in VMs ) is also benefitial. Despite the hiccups I have stumbled upon, I think that the project is great. Best Regards, Strahil Nikolov

2 1

Migrating self-HostedEngine from NFS to iSCSI
by Miha Verlic 21 May '19

21 May '19

Hello, I have a few questions regarding migration of HostedEngine. Currently I have a cluster of 3 oVirt 4.3.3 nodes, all three of them are capable of running HE and I can freely migrate HostedEngine and regular VMs between them. However I deployed HostedEngine storage on rather edgy NFS server and I would like to migrate it to iSCSI based storage with multipathing. Quite a few VMs are already running on cluster and are using iSCSI data storage. Documentation is rather chaotic and fragmented, but from what I gathered the path of migration is someting like: - place one host (#1), the "failover" host, into maintenance mode prior to backup - export configuration with engine-backup - set global maintenance mode on all hosts - install ovirt engine on that host (#1) (already installed, since this is HE capable host) - restore engine configuration using engine-backup - run engine-setup with new parameters regarding storage - after engine-setup, log into admin portal and remove old host (#1) - redeploy hosts #2 and #3 Last two steps are a bit confusing as I'm not sure how removing old failover host on which new HE is running would work. Also not understanding the part where hosts 2 and 3 are described as unrecoverable (but with running VMs, which I'd have to live migrate to other hosts - how, if they're not operational?). Few other things: - Should I first remove & re-add host #1 without HE already deployed on host? - Should I set global maintenance mode on all hosts before migration? I'm guessing this is required if I want to prevent HE being started on random host during transition... - Which host should be selected as SPM during the transition phase? - How can I configure iSCSI multipathing? Self-hosted engine documentation mentions Multipath Helper tool, however I cannot find any info about it. Is this tool freely available or only a part od RHEL subscription? - Can I configure existing iSCSI Domain which already hosts some VMs as HE storage? Or do I have to assign extra LUN/target exclusively for HE? Cheers -- Miha

2 1

Re: Scale out ovirt 4.3 (from 3 to 6 or 9 nodes) with hyperconverged setup and Gluster
by Strahil 21 May '19

21 May '19

Thanks for the clarification. It seems that my nvme (used by vdo) is not locked. I will check again before opening a bug. Best Regards, Strahil NikolovOn May 21, 2019 09:52, Sahina Bose <sabose(a)redhat.com> wrote: > > > > On Tue, May 21, 2019 at 2:36 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Hey Sahina, >> >> it seems that almost all of my devices are locked - just like Fred's. >> What exactly does it mean - I don't have any issues with my bricks/storage domains. > > > > If the devices show up as locked - it means the disk cannot be used to create a brick. This is when the disk either already has a filesystem or is in use. > But if the device is a clean device and it still shows up as locked - this could be a bug with how python-blivet/ vdsm reads this > > The code to check is implemented as > _canCreateBrick(device): > if not device or device.kids > 0 or device.format.type or \ > hasattr(device.format, 'mountpoint') or \ > device.type in ['cdrom', 'lvmvg', 'lvmthinpool', 'lvmlv', 'lvmthinlv']: > return False > return True > >> >> Best Regards, >> Strahil Nikolov >> >> В понеделник, 20 май 2019 г., 14:56:11 ч. Гринуич+3, Sahina Bose <sabose(a)redhat.com> написа: >> >> >> To scale existing volumes - you need to add bricks and run rebalance on the gluster volume so that data is correctly redistributed as Alex mentioned. >> We do support expanding existing volumes as the bug https://bugzilla.redhat.com/show_bug.cgi?id=1471031 has been fixed >> >> As to procedure to expand volumes: >> 1. Create bricks from UI - select Host -> Storage Device -> Storage device. Click on "Create Brick" >> If the device is shown as locked, make sure there's no signature on device. If multipath entries have been created for local devices, you can blacklist those devices in multipath.conf and restart multipath. >> (If you see device as locked even after you do this -please report back). >> 2. Expand volume using Volume -> Bricks -> Add Bricks, and select the 3 bricks created in previous step >> 3. Run Rebalance on the volume. Volume -> Rebalance. >> >> >> On Thu, May 16, 2019 at 2:48 PM Fred Rolland <frolland(a)redhat.com> wrote: >>> >>> Sahina, >>> Can someone from your team review the steps done by Adrian? >>> Thanks, >>> Freddy >>>

1 0

VM Windows on 4.2
by gpesoli＠it.iliad.com 21 May '19

21 May '19

Hi all, I still cannot install or migrate windows virtual machines (64bit and UEFI BIOS) on my oVirt platform (4.2.8.2-1), is it possible? On migration from ESXi, after converted them, VM doesn't want to start in anyway. Further I noticed that on new installation I can't install 64bit version of Windows, because seems that 64bit disk driver are not recognised.

5 6

Wrong disk size in UI after expanding iscsi direct LUN
by Bernhard Dick 21 May '19

21 May '19

Hi, I've extended the size of one of my direct iSCSI LUNs. The VM is seeing the new size but in the webinterface there is still the old size reported. Is there a way to update this information? I already took a look into the list but there are only reports regarding updating the size the VM sees. Best regards Bernhard

3 4

hosts becomes NonResponsive
by Jiří Sléžka 21 May '19

21 May '19

Hi, time to time one of our four ovirt hosts become NonResponsive. From engine point of view it looks this way (engine.log) 2019-05-21 13:10:30,261+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt03.net.slu.cz command Get Host Capabilities failed: Message timeout which can be caused by communication issues 2019-05-21 13:10:30,261+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [] Unable to RefreshCapabilities: VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues from host (which is reachable) it looks like (vdsm.log) 2019-05-21 13:10:27,154+0200 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList(options=None) from=internal, task_id=a1bebf2f-7070-4344-90b7-1d709ba94b5c (api:48) 2019-05-21 13:10:27,154+0200 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=a1bebf2f-7070-4344-90b7-1d709ba94b5c (api:54) 2019-05-21 13:10:27,155+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:709) 2019-05-21 13:10:31,245+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,39144 (api:48) 2019-05-21 13:10:31,247+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,39144 (api:54) 2019-05-21 13:10:31,249+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) hosts are latest CentOS7 (but old AMD Opteron HW), oVirt is 4.3.3.7-1.el7 I cannot track it down to network layer. We have 4 other RHV hosts on the same infrastructure and it works well. Some clues what is happening? Thanks in advance, Jiri Slezka

1 0

RHEL 8 Template Seal failed
by Vinícius Ferrão 21 May '19

21 May '19

Hello, I’m trying to seal a RHEL8 template but the operation is failing. Here’s the relevant information from engine.log: 2019-05-17 01:30:31,153-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-58) [91e1acd6-efc5-411b-8c76-970def4ebbbe] FINISH, GetHostJobsVDSCommand, return: {b80c0bbd-25b8-4007-9b91-376cb0a18e30=HostJobInfo:{id='b80c0bbd-25b8-4007-9b91-376cb0a18e30', type='virt', description='seal_vm', status='failed', progress='null', error='VDSError:{code='GeneralException', message='General Exception: ('Command [\'/usr/bin/virt-sysprep\', \'-a\', u\'/rhev/data-center/mnt/192.168.10.6:_mnt_pool0_ovirt_vm/d19456e4-0051-456e-b33c-57348a78c2e0/images/1ecdfbfc-1c22-452f-9a53-2159701549c8/f9de3eae-f475-451b-b587-f6a1405036e8\'] failed with rc=1 out=\'[ 0.0] Examining the guest ...\\nvirt-sysprep: warning: mount_options: mount exited with status 32: mount: \\nwrong fs type, bad option, bad superblock on /dev/mapper/rhel_rhel8-root,\\n missing codepage or helper program, or other error\\n\\n In some cases useful info is found in syslog - try\\n dmesg | tail or so. (ignored)\\nvirt-sysprep: warning: mount_options: mount: /boot: mount point is not a \\ndirectory (ignored)\\nvirt-sysprep: warning: mount_options: mount: /boot/efi: mount point is not \\na directory (ignored)\\n[ 17.9] Performing "abrt-data" ...\\n\' err="virt-sysprep: error: libguestfs error: glob_expand: glob_expand_stub: you \\nmust call \'mount\' first to mount the root filesystem\\n\\nIf reporting bugs, run virt-sysprep with debugging enabled and include the \\ncomplete output:\\n\\n virt-sysprep -v -x [...]\\n"',)'}'}}, log id: 1bbb34bf I’m not shure what’s wrong or missing. The VM image is using UEFI with Secure Boot, so standard UEFI partition is in place. Ive found something on bugzilla but does not seem to be related: https://bugzilla.redhat.com/show_bug.cgi?id=1671895 Thanks,

2 1

VM pools broken in 4.3
by Rik Theys 21 May '19

21 May '19

Hi, It seems VM pools are completely broken since our upgrade to 4.3. Is anybody else also experiencing this issue? Only a single instance from a pool can be used. Afterwards the pool becomes unusable due to a lock not being released. Once ovirt-engine is restarted, another (single) VM from a pool can be used. I've added my findings to bug 1462236, but I'm no longer sure the issue is the same as the one initially reported. When the first VM of a pool is started: 2019-05-14 13:26:46,058+02 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='d8a99676-d520-425e-9974-1b1efe6da8a5'}), log id: 2fb4f7f5 2019-05-14 13:26:46,058+02 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2fb4f7f5 2019-05-14 13:26:46,208+02 INFO [org.ovirt.engine.core.bll.VmPoolHandler] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d8a99676-d520-425e-9974-1b1efe6da8a5=VM]', sharedLocks=''}' -> it has acquired a lock (lock1) 2019-05-14 13:26:46,247+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[a5bed59c-d2fe-4fe4-bff7-52efe089ebd6=USER_VM_POOL]', sharedLocks=''}' -> it has acquired another lock (lock2) 2019-05-14 13:26:46,352+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Running command: AttachUserToVmFromPoolAndRunCommand internal: false. Entities affected : ID: 4c622213-e5f4-4032-8639-643174b698cc Type: VmPoolAction group VM_POOL_BASIC_OPERATIONS with role type USER 2019-05-14 13:26:46,393+02 INFO [org.ovirt.engine.core.bll.AddPermissionCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Running command: AddPermissionCommand internal: true. Entities affected : ID: d8a99676-d520-425e-9974-1b1efe6da8a5 Type: VMAction group MANIPULATE_PERMISSIONS with role type USER 2019-05-14 13:26:46,433+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Succeeded giving user 'a5bed59c-d2fe-4fe4-bff7-52efe089ebd6' permission to Vm 'd8a99676-d520-425e-9974-1b1efe6da8a5' 2019-05-14 13:26:46,608+02 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='d8a99676-d520-425e-9974-1b1efe6da8a5'}), log id: 67acc561 2019-05-14 13:26:46,608+02 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 67acc561 2019-05-14 13:26:46,719+02 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Running command:RunVmCommand internal: true. Entities affected : ID: d8a99676-d520-425e-9974-1b1efe6da8a5 Type: VMAction group RUN_VM with role type USER 2019-05-14 13:26:46,791+02 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{hostId='null', vmId='d8a99676-d520-425e-9974-1b1efe6da8a5', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@6db8c94d'}), log id: 2c110e4 2019-05-14 13:26:46,795+02 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] FINISH, UpdateVmDynamicDataVDSCommand, return: , log id: 2c110e4 2019-05-14 13:26:46,804+02 INFO [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] START,CreateVDSCommand( CreateVDSCommandParameters:{hostId='eec7ec2b-cae1-4bb9-b933-4dff47a70bdb', vmId='d8a99676-d520-425e-9974-1b1efe6da8a5', vm='VM [stud-c7-1]'}), log id: 71d599f2 2019-05-14 13:26:46,809+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] START, CreateBrokerVDSCommand(HostName = studvirt1, CreateVDSCommandParameters:{hostId='eec7ec2b-cae1-4bb9-b933-4dff47a70bdb', vmId='d8a99676-d520-425e-9974-1b1efe6da8a5', vm='VM [stud-c7-1]'}), log id: 3aa6b5ff 2019-05-14 13:26:46,836+02 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuildUtils] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Kernel FIPS - Guid: eec7ec2b-cae1-4bb9-b933-4dff47a70bdb fips: false 2019-05-14 13:26:46,903+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] VM <?xml version="1.0" encoding="UTF-8"?><domain type="kvm" xmlns:ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> [domain xml stripped] 2019-05-14 13:26:46,928+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] FINISH, CreateBrokerVDSCommand, return: , log id: 3aa6b5ff 2019-05-14 13:26:46,932+02 INFO [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] FINISH, CreateVDSCommand, return: WaitForLaunch, log id: 71d599f2 2019-05-14 13:26:46,932+02 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Lock freed to object 'EngineLock:{exclusiveLocks='[a5bed59c-d2fe-4fe4-bff7-52efe089ebd6=USER_VM_POOL]', sharedLocks=''}' -> it has released lock2 2019-05-14 13:26:47,004+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] EVENT_ID: VDS_INITIATED_RUN_VM(506), Trying to restart VM stud-c7-1 on Host studvirt1 2019-05-14 13:26:47,094+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] EVENT_ID: USER_ATTACH_USER_TO_VM_FROM_POOL(316), Attaching User u0045469 to VM stud-c7-1 in VM Pool stud-c7-? was initiated by u0045469(a)esat.kuleuven.be-authz <mailto:u0045469@esat.kuleuven.be-authz>. 2019-05-14 13:26:47,098+02 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Trying to release exclusive lock which does not exist, lock key: 'a5bed59c-d2fe-4fe4-bff7-52efe089ebd6USER_VM_POOL' -> it's trying to release the same lock2 as above and failing 2019-05-14 13:26:47,098+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-6) [e3c5745c-e593-4aed-ba67-b173808140e8] Lock freed to object 'EngineLock:{exclusiveLocks='[a5bed59c-d2fe-4fe4-bff7-52efe089ebd6=USER_VM_POOL]', sharedLocks=''}' -> now it's indicating that it released/freed the lock (lock2) 2019-05-14 13:26:48,518+02 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-3) [e3c5745c-e593-4aed-ba67-b173808140e8] Command 'AttachUserToVmFromPoolAndRun' id: '0148c91d-b053-4dc9-960c-f10bf0d3f343' child commands '[0470802d-09fa-4579-b95b-3d9603feff7b, 47dbfc58-3bae-4229-96eb-d1fc08911237]' executions were completed, status 'SUCCEEDED' 2019-05-14 13:26:49,584+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-14) [e3c5745c-e593-4aed-ba67-b173808140e8] Ending command 'org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand' successfully. 2019-05-14 13:26:49,650+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-14) [e3c5745c-e593-4aed-ba67-b173808140e8] EVENT_ID: USER_ATTACH_USER_TO_VM_FROM_POOL_FINISHED_SUCCESS(318), User u0045469 successfully attached to VM stud-c7-1 in VM Pool stud-c7-?. 2019-05-14 13:26:50,584+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-2) [] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), User <UNKNOWN> got disconnected from VM stud-c7-1. 2019-05-14 13:26:50,585+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-2) [] VM 'd8a99676-d520-425e-9974-1b1efe6da8a5'(stud-c7-1) moved from 'WaitForLaunch' --> 'PoweringUp' The first lock (on the pool itself) is never released. It seems this lock should also be released? Maybe it's this lock that should be released the second time (instead of the second lock twice)? When we start to launch another instance of the pool it fails: 2019-05-14 13:49:32,656+02 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-11) [55cc0796-4f53-49cd-8739-3b7e7dd2d95b] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='d8a99676-d520-425e-9974-1b1efe6da8a5'}), log id: 7db2f4fc 2019-05-14 13:49:32,656+02 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-11) [55cc0796-4f53-49cd-8739-3b7e7dd2d95b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7db2f4fc 2019-05-14 13:49:32,688+02 INFO [org.ovirt.engine.core.bll.VmPoolHandler] (default task-11) [55cc0796-4f53-49cd-8739-3b7e7dd2d95b] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[d8a99676-d520-425e-9974-1b1efe6da8a5=VM]', sharedLocks=''}' 2019-05-14 13:49:32,700+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-11) [55cc0796-4f53-49cd-8739-3b7e7dd2d95b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[a5bed59c-d2fe-4fe4-bff7-52efe089ebd6=USER_VM_POOL]', sharedLocks=''}' 2019-05-14 13:49:32,700+02 WARN [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-11) [55cc0796-4f53-49cd-8739-3b7e7dd2d95b] Validation of action 'AttachUserToVmFromPoolAndRun' failed for user u0045469(a)esat.kuleuven.be-authz <mailto:u0045469@esat.kuleuven.be-authz>. Reasons: VAR__ACTION__ALLOCATE_AND_RUN,VAR__TYPE__VM_FROM_VM_POOL,ACTION_TYPE_FAILED_NO_AVAILABLE_POOL_VMS 2019-05-14 13:49:32,700+02 INFO [org.ovirt.engine.core.bll.AttachUserToVmFromPoolAndRunCommand] (default task-11) [55cc0796-4f53-49cd-8739-3b7e7dd2d95b] Lock freed to object 'EngineLock:{exclusiveLocks='[a5bed59c-d2fe-4fe4-bff7-52efe089ebd6=USER_VM_POOL]', sharedLocks=''}' 2019-05-14 13:49:32,706+02 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-11) [] Operation Failed: [Cannot allocate and run VM from VM-Pool. There are no available VMs in the VM-Pool.] Regards, Rik -- Rik Theys System Engineer KU Leuven - Dept. Elektrotechniek (ESAT) Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee +32(0)16/32.11.07 ---------------------------------------------------------------- <<Any errors in spelling, tact or fact are transmission errors>>

3 7

Re: [ovirt-announce] Re: [ANN] oVirt 4.3.4 First Release Candidate is now available
by Strahil 21 May '19

21 May '19

On this one https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/h… We should have the following options: performance.quick-read=off performance.read-ahead=off performance.io-cache=off performance.stat-prefetch=off performance.low-prio-threads=32 network.remote-dio=enable cluster.eager-lock=enable cluster.quorum-type=auto cluster.server-quorum-type=server cluster.data-self-heal-algorithm=full cluster.locking-scheme=granular cluster.shd-max-threads=8 cluster.shd-wait-qlength=10000 features.shard=on user.cifs=off By the way the 'virt' gluster group disables 'cluster.choose-local' and I think it wasn't like that. Any reasons behind that , as I use it to speedup my reads, as local storage is faster than the network? Best Regards, Strahil NikolovOn May 19, 2019 09:36, Strahil <hunter86_bg(a)yahoo.com> wrote: > > OK, > > Can we summarize it: > 1. VDO must 'emulate512=true' > 2. 'network.remote-dio' should be off ? > > As per this: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/htm… > > We should have these: > > quick-read=off > read-ahead=off > io-cache=off > stat-prefetch=off > eager-lock=enable > remote-dio=on > quorum-type=auto > server-quorum-type=server > > I'm a little bit confused here. > > Best Regards, > Strahil Nikolov > > On May 19, 2019 07:44, Sahina Bose <sabose(a)redhat.com> wrote: >> >> >> >> On Sun, 19 May 2019 at 12:21 AM, Nir Soffer <nsoffer(a)redhat.com> wrote: >>> >>> On Fri, May 17, 2019 at 7:54 AM Gobinda Das <godas(a)redhat.com> wrote: >>>> >>>> From RHHI side default we are setting below volume options: >>>> >>>> { group: 'virt', >>>> storage.owner-uid: '36', >>>> storage.owner-gid: '36', >>>> network.ping-timeout: '30', >>>> performance.strict-o-direct: 'on', >>>> network.remote-dio: 'off' >>> >>> >>> According to the user reports, this configuration is not compatible with oVirt. >>> >>> Was this tested? >> >> >> Yes, this is set by default in all test configuration. We’re checking on the bug, but the error is likely when the underlying device does not support 512b writes. >> With network.remote-dio off gluster will ensure o-direct writes >>> >>> >>>> } >>>> >>>> >>>> On Fri, May 17, 2019 at 2:31 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >>>>> >>>>> Ok, setting 'gluster volume set data_fast4 network.remote-dio on' allowed me to create the storage domain without any issues. >>>>> I set it on all 4 new gluster volumes and the storage domains were successfully created. >>>>> >>>>> I have created bug for that: >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1711060 >>>>> >>>>> If someone else already opened - please ping me to mark this one as duplicate. >>>>> >>>>> Best Regards, >>>>> Strahil Nikolov >>>>> >>>>> >>>>> В четвъртък, 16 май 2019 г., 22:27:01 ч. Гринуич+3, Darrell Budic <budic(a)onholyground.com> написа: >>>>> >>>>> >>>>> On May 16, 2019, at 1:41 PM, Nir Soffer <nsoffer(a)redhat.com> wrote: >>>>> >>>>>> >>>>>> On Thu, May 16, 2019 at 8:38 PM Darrell Budic <budic(a)onholyground.com> wrote: >>>>>>> >>>>>>> I tried adding a new storage domain on my hyper converged test cluster running Ovirt 4.3.3.7 and gluster 6.1. I was able to create the new gluster volume fine, but it’s not able to add the gluster storage domain (as either a managed gluster volume or directly entering values). The created gluster volume mounts and looks fine from the CLI. Errors in VDSM log: >>>>>>> >>>>>> ... >>>>>>> >>>>>>> 2019-05-16 10:25:09,584-0500 ERROR (jsonrpc/5) [storage.fileSD] Underlying file system doesn't supportdirect IO (fileSD:110) >>>>>>> 2019-05-16 10:25:09,584-0500 INFO (jsonrpc/5) [vdsm.api] FINISH createStorageDomain error=Storage Domain target is unsupported: () from=::ffff:10.100.90.5,44732, flow_id=31d993dd, task_id=ecea28f3-60d4-476d-9ba8-b753b7c9940d (api:52) >>>>>> >>>>>> >>>>>> The direct I/O check has failed. >>>>>> >>>>>> >>>>>> So something is wrong in the files system. >>>>>> >>>>>> To confirm, you can try to do: >>>>>> >>>>>> dd if=/dev/zero of=/path/to/mountoint/test bs=4096 count=1 oflag=direct >>>>>> >>>>>> This will probably fail with: >>>>>> dd: failed to open '/path/to/mountoint/test': Invalid argument >>>>>> >>>>>> If it succeeds, but oVirt fail to connect to this domain, file a bug and we will investigate. >>>>>> >>>>>> Nir >>>>> >>>>> >>>>> Yep, it fails as expected. Just to check, it is working on pre-existing volumes, so I poked around at gluster settings for the new volume. It has network.remote-dio=off set on the new volume, but enabled on old volumes. After enabling it, I’m able to run the dd test: >>>>> >>>>> [root@boneyard mnt]# gluster vol set test network.remote-dio enable >>>>> volume set: success >>>>> [root@boneyard mnt]# dd if=/dev/zero of=testfile bs=4096 count=1 oflag=direct >>>>> 1+0 records in >>>>> 1+0 records out >>>>> 4096 bytes (4.1 kB) copied, 0.0018285 s, 2.2 MB/s >>>>> >>>>> I’m also able to add the storage domain in ovirt now. >>>>> >>>>> I see network.remote-dio=enable is part of the gluster virt group, so apparently it’s not getting set by ovirt duding the volume creation/optimze for storage? >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/OPBXHYOHZA4XR… >>>>> >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/B7K24XYG3M43C… >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> Thanks, >>>> Gobinda

3 2