August 2019 - Users - oVirt List Archives

Re: Where to configure iscsi initiator name?
by Gianluca Cecchi 22 Aug '19

22 Aug '19

On Wed, Aug 21, 2019 at 7:07 PM Dan Poltawski <dan.poltawski(a)tnp.net.uk> wrote: > When I added the first node a 'random' initiator name was generated of > form: > > # cat /etc/iscsi/initiatorname.iscsi > InitiatorName=iqn.1994-05.com.redhat:[RANDOM] > > Having attempted to add another node, this node has another initiator name > generated and can't access the storage. Is there a way to configure this > initiator name to a static value which will be configured when new nodes > get added to the cluster? Or is there some reason for this i'm missing? > > thanks, > > Dan > oVirt nodes are iSCSI clients and each node needs to have a different InitiatorName value. Eventually you can modify initiatorname of a node before running discovery against oVirt and rebooting it (or restarting iscsi services). You can edit the file or use the iscsi-iname command. It must be named with an iqn name in the format of iqn.YYYY-MM.reverse.domain.name:OptionalIdentifier Typically you configure your iSCSI storage array giving access to the lun to be used as a storage domain to all oVirt hosts initiators. Or you can use chap authentication or both Se also here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistent… HIH, Gianluca

1 0

Data Center Contending/Non responsive - No SPM
by Matthew B 21 Aug '19

21 Aug '19

Hello, I'm having a problem with oVirt - we're running 4.3 on CentOS 7: [root@ovirt ~]# rpm -q ovirt-engine; uname -r ovirt-engine-4.3.4.3-1.el7.noarch 3.10.0-957.21.2.el7.x86_64 Currently the Data Center alternates between Non responsive and contending status, and SPM selection fails. The error in the events tab is: VDSM compute5.domain command HSMGetTaskStatusVDS failed: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',) Full error: 2019-08-21 13:53:53,507-0700 ERROR (tasks/8) [storage.TaskManager.Task] (Task='3076bb8c-7462-4573-a832-337da478ae0e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in startSpm self._upgradePool(expectedDomVersion, __securityOverride=True) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 484, in _upgradePool str(targetDomVersion)) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1108, in _convertDomain targetFormat) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 447, in convert converter(repoPath, hostId, imageRepo, isMsd) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 405, in v5DomainConverter domain.convert_volumes_metadata(target_version) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 813, in convert_volumes_metadata for vol in self.iter_volumes(): File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 764, in iter_volumes yield self.produceVolume(img_id, vol_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 846, in produceVolume volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterVolume.py", line 45, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 817, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 71, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112, in validate self.validateVolumePath() File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 131, in validateVolumePath raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',) Most of our VMs are still running, but we can't start or restart any VMs. All of the domains show as down since SPM selection fails. Any thoughts? Thanks, -Matthew

1 0

Re: Where to configure iscsi initiator name?
by Vincent Royer 21 Aug '19

21 Aug '19

You have to add all the initiators for each node to the iscsi LUN. At least thats how I got it working. Then I think it's recommended to setup multipath. However take this with a grain because iscsi did not work well for me. On Wed, Aug 21, 2019, 10:08 AM Dan Poltawski <dan.poltawski(a)tnp.net.uk> wrote: > When I added the first node a 'random' initiator name was generated of > form: > > # cat /etc/iscsi/initiatorname.iscsi > InitiatorName=iqn.1994-05.com.redhat:[RANDOM] > > Having attempted to add another node, this node has another initiator name > generated and can't access the storage. Is there a way to configure this > initiator name to a static value which will be configured when new nodes > get added to the cluster? Or is there some reason for this i'm missing? > > thanks, > > Dan > ------------------------------ > > The Networking People (TNP) Limited. Registered office: Network House, > Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company > number: 07667393 > > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify the system manager. > This message contains confidential information and is intended only for the > individual named. If you are not the named addressee you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. If you are not the intended recipient > you are notified that disclosing, copying, distributing or taking any > action in reliance on the contents of this information is strictly > prohibited. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UKNN77NZJ4ZTB… >

1 0

hosted engine setup, iSCSI no LUNs shown
by billyburly＠gmail.com 21 Aug '19

21 Aug '19

I'm trying to setup the hosted engine on top of iSCSI storage. It successfully logs in and gets the target, however the process errors out claiming there are no LUNs. But if you look on the host, the disks were added to the system. [ INFO ] TASK [ovirt.hosted_engine_setup : iSCSI discover with REST API] [ INFO ] ok: [localhost] The following targets have been found: [1] iqn.2001-04.com.billdurr.durrnet.vm-int:vmdata TPGT: 1, portals: 192.168.47.10:3260 Please select a target (1) [1]: 1 [ INFO ] Getting iSCSI LUNs list ... [ INFO ] TASK [ovirt.hosted_engine_setup : Get iSCSI LUNs] [ INFO ] ok: [localhost] [ ERROR ] Cannot find any LUN on the selected target [ ERROR ] Unable to get target list Here's what the config in targetcli looks like [root@vm1 ~]# targetcli ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 2] | | o- p_iscsi_lun1 .............. [/dev/drbd0 (62.0GiB) write-thru activated] | | | o- alua ............................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ................... [ALUA state: Active/optimized] | | o- p_iscsi_lun2 ............. [/dev/drbd1 (310.6GiB) write-thru activated] | | o- alua ............................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ................... [ALUA state: Active/optimized] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] | o- iqn.2001-04.com.billdurr.durrnet.vm-int:vmdata ................ [TPGs: 1] | o- tpg1 .............................................. [gen-acls, no-auth] | o- acls ...................................................... [ACLs: 0] | o- luns ...................................................... [LUNs: 2] | | o- lun0 ......... [block/p_iscsi_lun1 (/dev/drbd0) (default_tg_pt_gp)] | | o- lun1 ......... [block/p_iscsi_lun2 (/dev/drbd1) (default_tg_pt_gp)] | o- portals ................................................ [Portals: 1] | o- 192.168.47.10:3260 ........................................... [OK] o- loopback ..................................................... [Targets: 0] o- srpt ......................................................... [Targets: 0] The two LUNs show up on the host after the hosted engine setup tries to enumerate the LUNs for the target [root@vm1 ~]# lsscsi [0:0:0:0] storage HP P420i 8.32 - [0:1:0:0] disk HP LOGICAL VOLUME 8.32 /dev/sda [0:1:0:1] disk HP LOGICAL VOLUME 8.32 /dev/sdb [0:1:0:2] disk HP LOGICAL VOLUME 8.32 /dev/sdc [11:0:0:0] disk LIO-ORG p_iscsi_lun1 4.0 /dev/sdd [11:0:0:1] disk LIO-ORG p_iscsi_lun2 4.0 /dev/sde

3 2

Reaction and timeouts to APD (All Paths Down)
by Gianluca Cecchi 21 Aug '19

21 Aug '19

Hello, similar questions/arguments took already place in the past, but I think it could be a good point to dig more if possible. I focus on block based storage and my environments have iSCSI based with multipath, connected to Equallogic. Currently I have set no_path_retry to 4, so I have a 20 seconds timeout (polling_interval=5). Sometimes some network planned activity (not under my domain) that should have no impact "doesn't go so well" and I could have longer delays (also 60-70 seconds) and so oVirt reactions where I see soft fencing, VMs going into paused state or "question mark" state... Comparing to vSphere, the same events don't apparently cause anything (and I see same lost paths events in datastore and ESXi host monitoring--> Events pane). This basically depends on APD default timeout that seems to be 140 seconds. This difference of behavior has the effect of showing an apparent better SLA of vSphere during these short time outages and a sort of show stopper for extending oVirt implementation.... Interesting vSphere KBs here: Storage device has entered the All Paths Down state (2032934) https://kb.vmware.com/s/article/2032934 Containing: "Note: By default, the APD timeout is set to 140 seconds." All Paths Down timeout for a storage device has expired (2032940) https://kb.vmware.com/s/article/2032940 Path redundancy to the storage device is degraded (1009555) https://kb.vmware.com/s/article/1009555 Storage device has recovered from the APD state (2032945) https://kb.vmware.com/s/article/2032945 So the question is what kind of real risks I have if I simulate that behavior and set a no_path_retry value so that polling_interval x no_path_retry = 140 (in default config of polling_interval=5 it would mean no_path_retry = 28) BTW: I also have an environment based on RHV 4.3.5 and iSCSI and in parallel I opened a case (02452597) for asking clarifications/chances remaining in supported configuration, so for logs and so on it could help going into it for Red Hat developers Thanks, Gianluca

1 0

Where to configure iscsi initiator name?
by Dan Poltawski 21 Aug '19

21 Aug '19

When I added the first node a 'random' initiator name was generated of form: # cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:[RANDOM] Having attempted to add another node, this node has another initiator name generated and can't access the storage. Is there a way to configure this initiator name to a static value which will be configured when new nodes get added to the cluster? Or is there some reason for this i'm missing? thanks, Dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

1 0

engine-setup is failing in fedora30
by Gobinda Das 21 Aug '19

21 Aug '19

Hi, I am facing issue to do engine-setup in fedora30. [godas@godas ovirt-engine]$ $HOME/ovirt-engine/bin/engine-setup [ INFO ] Stage: Initializing Setup was run under unprivileged user this will produce development installation do you wish to proceed? (Yes, No) [No]: Yes [ INFO ] Stage: Environment setup Configuration files: ['/home/gobinda/ovirt-engine/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /home/gobinda/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20190821121001-bkc6we.log Version: otopi-1.8.1_master (otopi-1.8.1-0.0.master.20190228094531.git1953bdc.fc30) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- --== PACKAGES ==-- --== NETWORK CONFIGURATION ==-- [WARNING] Failed to resolve godas.lab.eng.blr.redhat.com using DNS, it can be resolved only locally --== DATABASE CONFIGURATION ==-- --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation --== CONFIGURATION PREVIEW ==-- Default SAN wipe after delete : False Host FQDN : godas.lab.eng.blr.redhat.com Set up Cinderlib integration : False Engine database host : localhost Engine database port : 5432 Engine database secured connection : False Engine database host name validation : False Engine database name : engine Engine database user name : engine Engine installation : True PKI organization : lab.eng.blr.redhat.com Set up ovirt-provider-ovn : False Configure WebSocket Proxy : True Configure VMConsole Proxy : True Please confirm installation settings (OK, Cancel) [OK]: OK [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Upgrading CA [ INFO ] Configuring WebSocket Proxy [ INFO ] Backing up database localhost:engine to '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/backups/engine-20190821121010.0iip56rw.dump'. [ INFO ] Creating/refreshing Engine database schema Unregistering existing client registration info. [ INFO ] Generating post install configuration file '/home/gobinda/ovirt-engine/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ ERROR ] Failed to execute stage 'Misc configuration': a bytes-like object is required, not 'str' [ INFO ] Rolling back database schema [ INFO ] Clearing Engine database engine [ INFO ] Restoring Engine database engine [ INFO ] Restoring file '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/backups/engine-20190821121010.0iip56rw.dump' to database localhost:engine. [ ERROR ] Engine database rollback failed: cannot use a string pattern on a bytes-like object [ INFO ] Stage: Clean up Log file is located at /home/gobinda/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20190821121001-bkc6we.log [ INFO ] Generating answer file '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/setup/answers/20190821121039-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed I*t was working before but the moment i pulled latest master code the issue started.* -- Thanks, Gobinda

2 1

When I added the ovirt-node to the ovirt-engine through the web management interface, I get the following error: Host 172.16.41.100 installation failed. Failed to configure management network on the host. Host 172.16.41.100 does not comply with the cluster cluster-kvm networks, the following networks are missing on host: 'ovirtmgmt'
by wangyu13476969128＠126.com 21 Aug '19

21 Aug '19

When I added the ovirt-node to the ovirt-engine through the web management interface, I get the following error: Host 172.16.41.100 installation failed. Failed to configure management network on the host. Host 172.16.41.100 does not comply with the cluster cluster-kvm networks, the following networks are missing on host: 'ovirtmgmt' How can I solve this problem ?

3 12

VDSM ovirt-node-2 command Get Host Capabilities failed: Internal JSON-RPC error: {'reason': "invalid argument: KVM is not supported by '/usr/libexec/qemu-kvm' on this host"}
by wangyu13476969128＠126.com 20 Aug '19

20 Aug '19

The version of ovirt-engine is 4.3.5.5-1.el7 The version of ovirt-node-2 is 4.3.5.2-1.el7 When I add ovirt-node-2 to ovirt-engine ，it report: VDSM ovirt-node-2 command Get Host Capabilities failed: Internal JSON-RPC error: {'reason': "invalid argument: KVM is not supported by '/usr/libexec/qemu-kvm' on this host"} What 's the root cause of this problem ? How can I solve this problem ?

2 2

Hyperconverged setup ovirt 4.3.x using ansible?
by adrianquintero＠gmail.com 20 Aug '19

20 Aug '19

Hi, I am trying to do a Hyperconverged setup using ansible. So far I have been able to run a playbook to setup gluster but have not been able to identify how to install the Hosted-Engine VM using ansible and tie all the pieces together. Can someone point me in the right direction to deploy a hyperconverged environment using ansible? I have succesfully deployed ovirt 4.3.5 (Hyperconverged) using the web UI. thanks.

3 4

ovirt node install
by Staniforth, Paul 20 Aug '19

20 Aug '19

Hello, on the latest version of the oVirt-node install running nodectl info gives the error Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in <module> CliApplication() File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 200, in CliApplication return cmdmap.command(args) File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 118, in command return self.commands[command](**kwargs) File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 76, in info Info(self.imgbased, self.machine).write() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 46, in __init__ self._fetch_information() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 49, in _fetch_information self._get_bootloader_info() File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 62, in _get_bootloader_info bootinfo["entries"][k] = v.__dict__ AttributeError: 'list' object has no attribute '__dict__' Also what is the correct way to update from ovirt node 4.2 to 4.3? I used yum install https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-imag… I then did yum erase ovirt-release42 and rm /etc/yum.repos.d/ovirt-4.2* Regards, Paul S. To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

3 5

numa pinning for high performance
by Vincent Royer 20 Aug '19

20 Aug '19

Is there a good place to learn about cpu pinning and the various settings? I've built a test VM and am trying to understand - I've given it 2 cpus, 4 cores each, 1 thread per core. - The VM is pinned to a specific host with passthrough cpu enabled - numa pinning, first 4 cores on physical socket 1 and the next 4 on physical socket 2 - The host has two sockets each with 20 core cpus I would expect to see a load applied on 8 of the 40 cores on the host during testing. Instead it seems like 32 of 40 cores are working at 100%. What are the effects of this on the performance of the vm? [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png] [image: image.png]

3 3

Export as OVA fails
by anthonywest＠alfatron.com.au 20 Aug '19

20 Aug '19

Hi. We are running oVirt 4.3.4 with each of the hosts configured to use local storage. When I attempt to export a large virtual machine as an OVA the process begins but fails about half an hour later. On the oVirt Events page the following entries appear: Starting to export Vm Jessica as a Virtual Appliance Failed to export Vm Jessica as a Virtual Appliance to path /storage/Jessica.ova on Host Trueblood When I log onto the host Trueblood I see that a file called Jessica.ova.tmp has been created. The engine.log file contains the following errors: 2019-08-16 16:40:27,141+10 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Ansible playbook execution failed: Timeout occurred while executing Ansible playbook. 2019-08-16 16:40:27,142+10 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Failed to create OVA. Please check logs for more details: /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20190816161027-trueblood.alfatron.com.au-80ae4c6.log 2019-08-16 16:40:27,161+10 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Failed to create OVA file 2019-08-16 16:40:27,162+10 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] Command 'ExportVmToOva' id: '12e98544-67ea-45b0-adb8-c50c8c190ecf' with children [a6c04b63-fe81-4d13-a69f-87b8ac55ceaa] failed when attempting to perform the next operation, marking as 'ACTIVE' 2019-08-16 16:40:27,162+10 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-37) [80ae4c6] EngineException: ENGINE (Failed with error ENGINE and code 5001): org.ovirt.engine.core.common.errors.EngineException: EngineException: ENGINE (Failed with error ENGINE and code 5001) at org.ovirt.engine.core.bll.exportimport.ExportOvaCommand.createOva(ExportOvaCommand.java:117) [bll.jar:] at org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand.executeNextOperation(ExportVmToOvaCommand.java:224) [bll.jar:] at org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand.performNextOperation(ExportVmToOvaCommand.java:216) [bll.jar:] at org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:77) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_212] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2019-08-16 16:40:28,304+10 ERROR [org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [3f52073f-b7d2-4238-8ac1-4634678b11b4] Ending command 'org.ovirt.engine.core.bll.exportimport.ExportVmToOvaCommand' with failure. 2019-08-16 16:40:28,745+10 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [41e553bf] EVENT_ID: IMPORTEXPORT_EXPORT_VM_TO_OVA_FAILED(1,225), Failed to export Vm Jessica as a Virtual Appliance to path /storage/Jessica.ova on Host Trueblood Any suggestions/help on solving this would be much appreciated. Anthony

2 1

Issues with oVirt-Engine start - oVirt 4.3.4
by Vrgotic, Marko 19 Aug '19

19 Aug '19

Dear oVirt, While working on a procedure to get the NFS v4 mount from Netapp, working on oVIrt, following steps came out to be the way to go in regards of setting it up for oVIrt SHE and VM Guests: * mkdir /mnt/rhevstore * mount -t nfs 10.20.30.40:/ovirt_hosted_engine /mnt/rhevstore * chown -R 36.36 /mnt/rhevstore * chmod -R 755 /mnt/rhevstore * umount /mnt/rhevstore This works fine, and it needs to be executed on each Hypervisor, before its provisioned into oVirt. However, just today I have discovered that command chmod -R 755 /mnt/rhevstore, if executed on new to be added Hypervisor, after oVirt is already running, it brings the oVirt Engine into broken state. The moment I executed the above on 3rd Hypervisor, before provisioning it into oVirt, following occurred: * Engine threw following error: * 2019-08-19 13:16:31,425Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-82) [] Failed in 'SpmStatusVDS' method * Connection was lost: * packet_write_wait: Connection to 10.210.11.10 port 22: Broken pipe * And VDSM on SHE Hosting hypervisor started logging errors like: * 2019-08-19 15:00:52,340+0000 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) * 2019-08-19 15:00:53,865+0000 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/2 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f59442c3d50> at 0x7f59442c3b90> timeout=15, duration=225.00 at 0x7f592476df90> task#=578 at 0x7f59442ef910>, traceback: * File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap * self.__bootstrap_inner() * File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner * self.run() * File: "/usr/lib64/python2.7/threading.py", line 765, in run * self.__target(*self.__args, **self.__kwargs) * File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run * ret = func(*args, **kwargs) * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run * self._execute_task() * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task * task() * File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ * self._callable() * File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 186, in __call__ * self._func() * File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 481, in __call__ * stats = hostapi.get_stats(self._cif, self._samples.stats()) * File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 79, in get_stats * ret['haStats'] = _getHaInfo() * File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 177, in _getHaInfo * stats = instance.get_all_stats() * File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats * stats = broker.get_stats_from_storage() * File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in get_stats_from_storage * result = self._proxy.get_stats() * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ * return self.__send(self.__name, args) * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request * verbose=self.__verbose * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request * return self.single_request(host, handler, request_body, verbose) * File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request * response = h.getresponse(buffering=True) * File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse * response.begin() * File: "/usr/lib64/python2.7/httplib.py", line 444, in begin * version, status, reason = self._read_status() * File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status * line = self.fp.readline(_MAXLINE + 1) * File: "/usr/lib64/python2.7/socket.py", line 476, in readline * data = self._sock.recv(self._rbufsize) (executor:363) * 2019-08-19 15:00:54,103+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) I am unable to boot the Engine VM – it end up in Status ForceStop Hosted-engine –vm-status shows: [root@ovirt-sj-02 ~]# hosted-engine --vm-status The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. But, storage is mounted and reachable, as well as ovirt-ha-agent running: [root@ovirt-sj-02 ~]# systemctl status ovirt-ha-agent ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-08-19 14:57:07 UTC; 23s ago Main PID: 43388 (ovirt-ha-agent) Tasks: 2 CGroup: /system.slice/ovirt-ha-agent.service └─43388 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent Can somebody help me with what to do? — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

1 2

Hosted Engine on seperate L2 network from nodes?
by Dan Poltawski 19 Aug '19

19 Aug '19

For some security requirements, I’ve been asked if it’s possible to segregate the hosted engine from the physical nodes, with specific firewalling for access to do node/ storage operations (I’m using managed block storage). Is this an approach others us, or is it better practice and just ensure the nodes and engine are all sharing the same network? Thanks, dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

2 1

Adding VLANs to a single-host, self-hosted-engine oVirt deployment?
by Derek Atkins 16 Aug '19

16 Aug '19

HI, I've got a single oVirt host running a self-contained hosted-engine deployment. When I set it up I did not use VLANs in my network. I am in the process of moving my equipment, and in part of this move I would like to introduce VLANs into my network infrastructure. The documentation seems to imply that to add virtual networks and/or VLANs to a host that I need to put it into maintenance mode, configure it in the engine, and then resync the network. However, I don't think I can do that with a single-host environment. If I put the host into local maint mode, it will try to offload all my VMs, including the engine, which obviously it cannot do because there is no other host to migrate them to. So what's the approach to add VLANs in this situation? I should add that this system started at 4.0, and I'm still only running 4.1 (although I do plan to upgrade to 4.2 as part of this move). I'm hesitant to upgrade further because of the impending removal of SDK-3 -- I am depending on a script that uses ovirt-shell which I keep being told is going away. If ovirt-shell is still in 4.3 then I might consider upgrading to that as well. :) Thanks. -derek -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

2 3

Can't add a pre-existing (storage-only) gluster member as a compute-also node using self-hosted hyper-convergent engine
by thomas＠hoberg.net 16 Aug '19

16 Aug '19

Intro: Trying to add oVirt self-hosted engine to a pre-existing set of system already using Gluster storage seems to fail, because ‘hosted engine wizard’ seems to create its own peer pool for managed hosts, instead of joining the one given for storage. Context: I am testing on a set of four Atom Goldmont+ boxes, which are silent/low-power, cheap, fast-enough for edge workloads and even take 32GB of RAM these days. But for various, sometimes even good reasons, they are not a mainline platform and I attributed many problems I faced to the niche hardware--sometimes correctly. Because the three-node hyperconverged setup has very exacting requirements, hard to meet in pre-existing machines (and in my case many initial failures with too little insight), I created the storage gluster separately first and then used the “hosted engine” wizard to set up the hosted engine on one of the Atom nodes. I used CentOS7 (fresh install and latest updates) on the primary nodes, not the oVirt node image, because some of my targets are big HPC machines, that are supposed to run oVirt for support services in a small niche, while Nvidia-Docker/SLURM workloads dominate. I assumed that if I were to split the storage and the orchestration setup into distinct steps, it would give me both better insight and flexibility to expand/transform the storage and the compute without losing any of the self-hosted hyperconvergeance comfort. Problem: I had all kinds of problem to get the hosted engine to run all the way through on the Atoms, it typically stopped just shy of the final launch as a VM on the Gluster storage. I eventually stumbled across this message from Yedidyah Bar David: https://lists.ovirt.org/pipermail/users/2018-March/087923.html I then had a look at the engine database and found that indeed the compute nodes were all in a separate gluster newly created by the hosted-engine setup and evidently used for cluster synchronization, instead of joining the pool already used for the bulk of the storage. I don’t know if this would be considered an error that needs fixing, an issue that can be avoided using a manual configuration, or something else. I believe it could use some highlighting in the documentation.

1 2

Hyperconverged on a self-made Gluster: Considered normal or 'way-off'?
by thomas＠hoberg.net 16 Aug '19

16 Aug '19

I use three very low power target nodes (J5005 Intel Atoms, 24GB RAM, 250GB SSD) to experiment with hyperconverged oVirt, and some bigger machines to act as pure compute nodes, preferably CentOS based, potentially oVirt Node image based. I've used both CentOS7 and oVirt Node image bases (all latest versions and patches) and typically I use either a single partition and xfs root file system to avoid partitioning the constricted space on CentOS or the recommended partition layout with the oVirt node images, which still consumes the single primary SSD /dev/sda. In both cases I don't have another device or partition left for the three-node hyperconverged wizard's ansible scripts to play with. And since I want to understand better what is going on under the hood in any case I tried to separate the Gluster setup and the hosted-engine setup. Question: Is the impact of that decision perhaps far bigger than I imagined? So I went with the alternative A Hosted-Engine wizard, where the storage was already provisioned and I had set up 2+1 Gluster storage using bricks right on the xfs, assuming I would achieve something very similar to what the three node wizard would simplify, except without the LVM/VDO layer all done with Ansible. But I am running into all kinds of problems and what continues to irritate me is the note in the manuals, that clusters need to be either 'compute' or 'storage' but cannot be both--yet that's the very point of hyperconvergence... So basically I am wondering whether at this point there is a strict bifurifcation of oVirt between the hyperconverged variant and the 'Hosted-Engine' on existing storage, without any middle ground. Or is it ok to do a two-phased approach, where you separate out the Gluster aspect and the Hosted-Engine *but with physical servers that are doing both storage and compute*? Also, I am wondering about the 'Enable Virt Service' and 'Enable Gluster Service' tick boxes in the Cluster property pages: For my 'hosted engine' wizard generated Cluster 'gluster' was unticked, even if it did run on Gluster storage. And while ticking 'gluster' in addition had no visible effect, unticking either one of the two is now impossible. I could not find any documentation on the significance or effect of these boxes. While the prepared three node replicated storage Gluster for the /engine was running on CentOS nodes, the Hosted-Engine VM was then created on additional oVirt Node hosts using that Gluster storage (trying to do the same on CentOS nodes always fails, separate topic), with the assumption that I could then push the hosted-engine VM back to the gluster storage nodes and get rid of the temporary compute-only host used for hosted engine setup. I then tried to add the Gluster nodes as hosts (and hosted-engine backup nodes), that also failed. Whether it failed because they were CentOS based or because they were already used for storage (and there is this hidden exclusivity), I would like to have some clarity on. So to repeat once again my main question: Is the (one or three) node hyperconverged oVirt setup a completely different thing from the hosted-engine on Gluster? Are you supposed to be able to add extra compute hosts, storage etc. to be managed by a three-node HC based hosted-engine? Are you supposed to be able to expand the three-node HC also in terms of storage, if perhaps currently only in quorum-maintaining multiples? Is oVirt currently actually strictly mandating 100% segregation between compute and storage unless you use the wholly contained single/triple HC setups?

1 0

Disk Allocation policy changes after a snapshot
by Kevin Doyle 16 Aug '19

16 Aug '19

Hi I have linux VM with 2 disks one for OS is sparse the other is for a Database and is Preallocated. When I take a snapshot of the VM both disks change to sparse policy but the disks in the snapshot are 1 sparse and 1 allocated. Before the snapshot the VM was running fine, now it crashes when data is written to the database. When I delete the snapshot the disks go back to 1 sparse and 1 allocated. Has anyone else seen this happen. Ovirt is 4.3.2.1-1.el7 and it is running on a hostedEngine Many thanks Kevin

4 5

failed import ova vm
by David David 16 Aug '19

16 Aug '19

hi cant import vmware OVA Engine: 4.3.5.5-1.el7 OS Version: RHEL - 7 - 6.1810.2.el7.centos Kernel Version: 3.10.0 - 957.27.2.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.7.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.12 VDSM Version: vdsm-4.30.24-1.el7 WebUI -> Virtual Machines -> Import (select vmware OVA, select OVAs path, Load) - OK(details) - OK vdsm.log, engine.log in attachment

2 1

Inconsistent metadata on VM's disks
by Nardus Geldenhuys 15 Aug '19

15 Aug '19

Hi There Hope you are all well. We got this weird issue at the moment. Let me explain. We are on ovirt 4.3.5. We use Centos 7.6 and when we do an update on the VM's the VM's can not boot into multi user mode. It complains that the root mount can't be found. When you boot using ISO and rescue mode you can see the disks. When you chroot into the VM and run pvscan twice your VM can be rebooted and it is fixed. We don't know if this is Centos based or maybe something on ovirt. Screenshots below. Any help would be appreciated. [image: Screenshot from 2019-08-11 18-15-14.png] [image: Screenshot from 2019-08-12 15-10-53.png] [image: Screenshot from 2019-08-12 14-18-33.png] [image: Screenshot from 2019-08-09 21-54-44.png]

2 1

Looking to install additional packages on Ovirt Node
by krahmani＠ieee.org 14 Aug '19

14 Aug '19

Hello all, Hope everyone is doing well! I would simply like to ask how it would be possible to add additional packages to the Ovirt Centos node. I followed the instructions here: https://serverfault.com/questions/927652/is-it-ok-to-add-the-epel-repo-to-a… but I am still receiving an error that the "rear" package cannot be found (the package I am trying to install, ofcourse). Please help, I appreciate taking the time to read this. Thank you!

2 1

Re: VM --- is not responding.
by Strahil 14 Aug '19

14 Aug '19

Would you check the health status of the controllers : hpssacli ctrl all show status Best Regards, Strahil NikolovOn Aug 11, 2019 09:55, Edoardo Mazza <edo7411(a)gmail.com> wrote: > > The hosts are 3 ProLiant DL380 Gen10, 2 hosts with HPE Smart Array P816i-a SR Gen10 like controller and the other host with > HPE Smart Array P408i-a SR Gen10. The storage for ovirt enviroment is gluster and the last host is the arbiter in the gluster enviroment. > The S.M.A.R.T. healt status is ok for all host > Edoardo > > > > > > Il giorno gio 8 ago 2019 alle ore 16:19 Sandro Bonazzola <sbonazzo(a)redhat.com> ha scritto: >> >> >> >> Il giorno gio 8 ago 2019 alle ore 11:19 Edoardo Mazza <edo7411(a)gmail.com> ha scritto: >>> >>> Hi all, >>> It is more days that for same vm I received this error, but I don't underdand why. >>> The traffic of the virtual machine is not excessive, cpu and ram to, but for few minutes the vm is not responding. and in the messages log file of the vm I received the error under, yo can help me? >>> thanks >> >> >> can you check the S.M.A.R.T. health status of the disks? >> >> >>> >>> Edoardo >>> kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:26227] >>> Aug 8 02:51:11 vmmysql kernel: Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ >>> ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_ >>> nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con >>> ntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_c >>> odec_generic iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul >>> glue_helper ablk_helper snd_hwdep cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore virtio_rng sg virtio_balloon >>> i2c_piix4 parport_pc parport joydev pcspkr ip_tables xfs libcrc32c sd_mod >>> Aug 8 02:51:14 vmmysql kernel: crc_t10dif crct10dif_generic sr_mod cdrom virtio_net virtio_console virtio_scsi ata_generic p >>> ata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw qxl floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod >>> Aug 8 02:51:14 vmmysql kernel: CPU: 2 PID: 26227 Comm: kworker/2:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.1.el7.x86_64 #1 >>> Aug 8 02:51:14 vmmysql kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 >>> Aug 8 02:51:14 vmmysql kernel: Workqueue: events_freezable disk_events_workfn >>> Aug 8 02:51:14 vmmysql kernel: task: ffff9e25b6609040 ti: ffff9e27b1610000 task.ti: ffff9e27b1610000 >>> Aug 8 02:51:14 vmmysql kernel: RIP: 0010:[<ffffffffb8b6b355>] [<ffffffffb8b6b355>] _raw_spin_unlock_irqrestore+0x15/0x20 >>> Aug 8 02:51:14 vmmysql kernel: RSP: 0000:ffff9e27b1613a68 EFLAGS: 00000286 >>> Aug 8 02:51:14 vmmysql kernel: RAX: 0000000000000001 RBX: ffff9e27b1613a10 RCX: ffff9e27b72a3d05 >>> Aug 8 02:51:14 vmmysql kernel: RDX: ffff9e27b729a420 RSI: 0000000000000286 RDI: 0000000000000286 >>> Aug 8 02:51:14 vmmysql kernel: RBP: ffff9e27b1613a68 R08: 0000000000000001 R09: ffff9e25b67fc198 >>> Aug 8 02:51:14 vmmysql kernel: R10: ffff9e27b45bd8d8 R11: 0000000000000000 R12: ffff9e25b67fde80 >>> Aug 8 02:51:14 vmmysql kernel: R13: ffff9e25b67fc000 R14: ffff9e25b67fc158 R15: ffffffffc032f8e0 >>> Aug 8 02:51:14 vmmysql kernel: FS: 0000000000000000(0000) GS:ffff9e27b7280000(0000) knlGS:0000000000000000 >>> Aug 8 02:51:14 vmmysql kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> Aug 8 02:51:14 vmmysql kernel: CR2: 00007f0c9e9b6008 CR3: 00

2 1

Either allow 2 CD-ROM's or selectable *.vfd Floppy Images from Storage via Run Once other than from deprcated ISO-Storage
by Ralf Schenk 13 Aug '19

13 Aug '19

Hello, when Installing Windows VM's on Ovirt we need either 2 CD-ROM's attached as ISO Files (Installer ISO and Virtio-Win-ISO) to be able to install to Virtio-(SCSI)-Disks. In Ovirt 4.3.4 it is not possible to attach 2 CD-ROM's to a VM. So we have to use Floppy Images (virtio-win-*.vfd) attached to install drivers within Installer. We need to use "Run Once" to attach flopppy disks. There are only *.vfd selectable which are located on ISO-Storage.Domain, which will be deprecated now or then. -> We won't be able to install Windows VM's from unmodified ISO Installer-CD's without ISO Storage Domain or making *.vfd Files selectable via "Run Once" When will that be available... ? Bye -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------

6 5

Re: oVirt 4.3.5 potential issue with NFS storage
by Shani Leviim 13 Aug '19

13 Aug '19

Basically, I meant to verify the access by ssh, but I want to verify something following your detailed reply: According to [1], in order to set a NetApp NFS server, the required steps should look like this: # mount NetApp_NFS:/path/to/export /mnt # chown -R 36.36 /mnt # chmod -R 755 /mnt # umount /mnt Which is quite similar to the steps you've mentioned, except the last step of unmounting: Unmount the 10.214.13.64:/ovirt_production I think that you had to unmount /mnt/rhevstore instead. Can you please verify? [1] https://access.redhat.com/solutions/660143 *Regards,* *Shani Leviim* On Sun, Aug 11, 2019 at 10:57 PM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> wrote: > Hi Shani, > > Thank you for your reply, but > How do I do that? > Reason why I am asking is following: > Hosts 2,3,4 do not have that issue. Host 1 and 5 do. > What I learned previously is that when using Netapp based NFS, which we > are, it’s required to before provisioning SHE and/or just adding a Host to > a pool, it’s required to execute following steps: > > Create random dir on a host: > - mkdir /mnt/rhevstore > Mount netapp volume to the dir > - mount -o sec=sys -t nfs 10.214.13.64:/ovirt_production /mnt/rhevstore > Set ownership to vdsm:kvm (36:36): > - chown -R vdsm:kvm /mnt/rhevstore/* > Unmount the 10.214.13.64:/ovirt_production > > I do not expect the above ownership actions need to be done initially on > each host, before starting the deployment, otherwise it would be > practically impossible to expand the Host pool. > > All 5 hosts are provisioned in same way. How? I am using foreman to > provision these servers, so they are built of same kickstart hostgroup > template. > > I even installed ovirt-hosted-engine-setup package to make sure all > required packages, users and groups are in place before adding host to > oVirt via UI or Ansible. > > Is it possible that we if I am already using or heavily using the > mentioned volume via Hosts already added to oVirt pool, that ownership > actions executed,on host about to be added to the pool, will fail to > complete setting ownership on all required files on the volume? > > To repeat the question above: How do I make sure Host can read metadata > file of the storage volume? > > Kindly awaiting your reply. > > > All best, > Marko Vrgotic > Sent from my iPhone > > On 11 Aug 2019, at 01:19, Shani Leviim <sleviim(a)redhat.com> wrote: > > Hi Marko, > Is seems that there's a connectivity problem with host 10.210.13.64. > Can you please make sure the metadata under > /rhev/data-center/mnt/10.210.13.64:_ovirt__production/6effda5e-1a0d-4312-bf93-d97fa9eb5aee/dom_md/metadata > is accessible? > > > *Regards, * > > *Shani Leviim * > > > On Sat, Aug 10, 2019 at 2:57 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> > wrote: > >> Log files from ovirt engine and ovirt-sj-05 vdsm attached. >> >> >> >> Its related to host named: ovirt-sj-05.ictv.com >> >> >> >> Kindly awaiting your reply. >> >> >> >> >> >> — — — >> Met vriendelijke groet / Kind regards, >> >> *Marko Vrgotic* >> >> >> >> >> >> >> >> *From: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> >> *Date: *Thursday, 8 August 2019 at 17:02 >> *To: *Shani Leviim <sleviim(a)redhat.com> >> *Cc: *"users(a)ovirt.org" <users(a)ovirt.org> >> *Subject: *Re: [ovirt-users] Re: oVirt 4.3.5 potential issue with NFS >> storage >> >> >> >> Hey Shanii, >> >> >> >> Thank you for the reply. >> >> Sure, I will attach the full logs asap. >> >> What do you mean by “flow you are doing”? >> >> >> >> Kindly awaiting your reply. >> >> >> >> Marko Vrgotic >> >> >> >> *From: *Shani Leviim <sleviim(a)redhat.com> >> *Date: *Thursday, 8 August 2019 at 00:01 >> *To: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> >> *Cc: *"users(a)ovirt.org" <users(a)ovirt.org> >> *Subject: *Re: [ovirt-users] Re: oVirt 4.3.5 potential issue with NFS >> storage >> >> >> >> Hi, >> >> Can you please clarify the flow you're doing? >> >> Also, can you please attach full vdsm and engine logs? >> >> >> *Regards,* >> >> *Shani Leviim* >> >> >> >> >> >> On Thu, Aug 8, 2019 at 6:25 AM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> >> wrote: >> >> Log line form VDSM: >> >> >> >> “[root@ovirt-sj-05 ~]# tail -f /var/log/vdsm/vdsm.log | grep WARN >> >> 2019-08-07 09:40:03,556-0700 WARN (check/loop) [storage.check] Checker >> u'/rhev/data-center/mnt/10.210.13.64:_ovirt__production/bda97276-a399-448f-9113-017972f6b55a/dom_md/metadata' >> is blocked for 20.00 seconds (check:282) >> >> 2019-08-07 09:40:47,132-0700 WARN (monitor/bda9727) [storage.Monitor] >> Host id for domain bda97276-a399-448f-9113-017972f6b55a was released (id: >> 5) (monitor:445) >> >> 2019-08-07 09:44:53,564-0700 WARN (check/loop) [storage.check] Checker >> u'/rhev/data-center/mnt/10.210.13.64:_ovirt__production/bda97276-a399-448f-9113-017972f6b55a/dom_md/metadata' >> is blocked for 20.00 seconds (check:282) >> >> 2019-08-07 09:46:38,604-0700 WARN (monitor/bda9727) [storage.Monitor] >> Host id for domain bda97276-a399-448f-9113-017972f6b55a was released (id: >> 5) (monitor:445)” >> >> >> >> >> >> >> >> *From: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> >> *Date: *Wednesday, 7 August 2019 at 09:09 >> *To: *"users(a)ovirt.org" <users(a)ovirt.org> >> *Subject: *oVirt 4.3.5 potential issue with NFS storage >> >> >> >> Dear oVIrt, >> >> >> >> This is my third oVirt platform in the company, but first time I am >> seeing following logs: >> >> >> >> “2019-08-07 16:00:16,099Z INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-51) [1b85e637] Lock freed >> to object >> 'EngineLock:{exclusiveLocks='[2350ee82-94ed-4f90-9366-451e0104d1d6=PROVIDER]', >> sharedLocks=''}' >> >> 2019-08-07 16:00:25,618Z WARN >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37723) [] domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' in problem >> 'PROBLEMATIC'. vds: 'ovirt-sj-05.ictv.com' >> >> 2019-08-07 16:00:40,630Z INFO >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37735) [] Domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' recovered from >> problem. vds: 'ovirt-sj-05.ictv.com' >> >> 2019-08-07 16:00:40,652Z INFO >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37737) [] Domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' recovered from >> problem. vds: 'ovirt-sj-01.ictv.com' >> >> 2019-08-07 16:00:40,652Z INFO >> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] >> (EE-ManagedThreadFactory-engine-Thread-37737) [] Domain >> 'bda97276-a399-448f-9113-017972f6b55a:ovirt_production' has recovered from >> problem. No active host in the DC is reporting it as problematic, so >> clearing the domain recovery timer.” >> >> >> >> Can you help me understanding why is this being reported? >> >> >> >> This setup is: >> >> >> >> 5HOSTS, 3 in HA >> >> SelfHostedEngine >> >> Version 4.3.5 >> >> NFS based Netapp storage, version 4.1 >> >> “10.210.13.64:/ovirt_hosted_engine on /rhev/data-center/mnt/10.210.13.64:_ovirt__hosted__engine >> type nfs4 >> (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=10.210.11.14,local_lock=none,addr=10.210.13.64) >> >> >> >> 10.210.13.64:/ovirt_production on /rhev/data-center/mnt/10.210.13.64:_ovirt__production >> type nfs4 >> (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=10.210.11.14,local_lock=none,addr=10.210.13.64) >> >> tmpfs on /run/user/0 type tmpfs >> (rw,nosuid,nodev,relatime,seclabel,size=9878396k,mode=700)” >> >> >> >> First mount is SHE dedicated storage. >> >> Second mount “ovirt_produciton” is for other VM Guests. >> >> >> >> Kindly awaiting your reply. >> >> >> >> Marko Vrgotic >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/ICRKHD3GXTPQE… >> >>

2 1