Sometimes paused due to unknown storage error on gluster
by Gianluca Cecchi
Hello,
having deployed oVirt 4.3.9 single host HCI with Gluster, I see some times
VM going into paused state for the error above and needing to manually run
it (sometimes this resumal operation fails).
Actually it only happened with empty disk (thin provisioned) and sudden
high I/O during the initial phase of install of the OS; it didn't happened
then during normal operaton (even with 600MB/s of throughput).
I suspect something related to metadata extension not able to be in pair
with the speed of the physical disk growing.... similar to what happens for
block based storage domains where the LVM layer has to extend the logical
volume representing the virtual disk
My real world reproduction of the error is during install of OCP 4.3.8
master node, when Red Hat Cores OS boots from network and wipes the disk
and I think then transfer an image, so doing high immediate I/O.
The VM used as master node has been created with a 120Gb thin provisioned
disk (virtio-scsi type) and starts with disk just initialized and empty,
going through PXE install.
I get this line inside events for the VM
Mar 27, 2020, 12:35:23 AM VM master01 has been paused due to unknown
storage error.
Here logs around the time frame above:
- engine.log
https://drive.google.com/file/d/1zpNo5IgFVTAlKXHiAMTL-uvaoXSNMVRO/view?us...
- vdsm.log
https://drive.google.com/file/d/1v8kR0N6PdHBJ5hYzEYKl4-m7v1Lb_cYX/view?us...
Any suggestions?
The disk of the VM is on vmstore storage domain and its gluster volume
settings are:
[root@ovirt tmp]# gluster volume info vmstore
Volume Name: vmstore
Type: Distribute
Volume ID: a6203d77-3b9d-49f9-94c5-9e30562959c4
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: ovirtst.mydomain.storage:/gluster_bricks/vmstore/vmstore
Options Reconfigured:
performance.low-prio-threads: 32
storage.owner-gid: 36
performance.read-ahead: off
user.cifs: off
storage.owner-uid: 36
performance.io-cache: off
performance.quick-read: off
network.ping-timeout: 30
features.shard: on
network.remote-dio: off
cluster.eager-lock: enable
performance.strict-o-direct: on
transport.address-family: inet
nfs.disable: on
[root@ovirt tmp]#
What about config above, related to eventual optimizations to be done based
on having single host?
And comparing with the virt group of options:
[root@ovirt tmp]# cat /var/lib/glusterd/groups/virt
performance.quick-read=off
performance.read-ahead=off
performance.io-cache=off
performance.low-prio-threads=32
network.remote-dio=enable
cluster.eager-lock=enable
cluster.quorum-type=auto
cluster.server-quorum-type=server
cluster.data-self-heal-algorithm=full
cluster.locking-scheme=granular
cluster.shd-max-threads=8
cluster.shd-wait-qlength=10000
features.shard=on
user.cifs=off
cluster.choose-local=off
client.event-threads=4
server.event-threads=4
performance.client-io-threads=on
[root@ovirt tmp]#
?
Thanks Gianluca
2 years, 11 months
DR on hyperconverged deployment
by wodel youchi
Hi,
Need to understand somethings about DR on oVirt-HI
- What does mean : Scheduling regular backups using geo-replication
(point 3.3.4 RHHI 1.7 Doc Maintaining RHHI) :
- Does this mean creating a check-point?
- If yes, does this mean that the geo-replication process will sync
data up to that check-point and then stops the synchronization,
then repeat
the same cycle the day after? does this mean that the minimum RPO is one
day?
- I created a snapshot of a VM on the source Manager, I synced the
volume then I executed a DR, The VM was started on the Target Manager but
the VM didn't have its snapshot, any idea???
Regards, be safe.
2 years, 11 months
Ovirt and Dell Compellent in ISCSI
by dalmasso＠cines.fr
hi all,
we use ovirt 4.3 on dell server r640 runing centos 7.7 and a storage bay Dell Compellent SCv3020 in ISCSI.
We use two 10gb interfaces for iSCSI connection on each dell server.
If we configure ISCSI connection directly from web IU, we can’t specify the two physical ethernet interface , and there are missing path . (only 4 path on 8)
So, on the shell of hypervisor we use this commands for configure the connections :
iscsiadm -m iface -I em1 --op=new # 1st ethernet interface
iscsiadm -m iface -I p3p1 --op=new # 2d ethernet interface
iscsiadm -m discovery -t sendtargets -p xx.xx.xx.xx
iscsiadm -m node -o show
iscsiadm -m node --login
after this, on the web IU we can connect our LUN with all path.
Also, I don’t understand how to configure multipath in the web UI . By defaut the configuration is in failover :
multipath -ll :
36000d3100457e4000000000000000005 dm-3 COMPELNT,Compellent Vol
size=500G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 23:0:0:1 sdb 8:16 active ready running
|- 24:0:0:1 sdd 8:48 active ready running
|- 25:0:0:1 sdc 8:32 active ready running
|- 26:0:0:1 sde 8:64 active ready running
|- 31:0:0:1 sdf 8:80 active ready running
|- 32:0:0:1 sdg 8:96 active ready running
|- 33:0:0:1 sdh 8:112 active ready running
|- 34:0:0:1 sdi 8:128 active ready running
I think round robind or another configuration will be more performent.
So can we made this configuration , select physical interface and configure multipath in web UI ? for easyly maintenance and adding other server ?
Thank you.
Sylvain.
2 years, 11 months
oVirt 4.4.0 Beta release is now available for testing
by Sandro Bonazzola
oVirt 4.4.0 Beta release is now available for testing
The oVirt Project is excited to announce the availability of the beta
release of oVirt 4.4.0 for testing, as of March 27th, 2020
This release unleashes an altogether more powerful and flexible open source
virtualization solution that encompasses hundreds of individual changes and
a wide range of enhancements across the engine, storage, network, user
interface, and analytics on top of oVirt 4.3.
Important notes before you try it
Please note this is a Beta release.
The oVirt Project makes no guarantees as to its suitability or usefulness.
This pre-release must not to be used in production.
In particular, please note that upgrades from 4.3 and future upgrades from
this beta to the final 4.4 release from this version are not supported.
Some of the features included in oVirt 4.4.0 Beta require content that will
be available in CentOS Linux 8.2 which are currently included in Red Hat
Enterprise Linux 8.2 beta. If you want to have a better experience you can
test oVirt 4.4.0 Beta on Red Hat Enterprise Linux 8.2 beta.
Known Issues
-
ovirt-imageio development is still in progress. In this beta you can’t
upload images to data domains. You can still copy iso images into the
deprecated ISO domain for installing VMs.
Installation instructions
For the engine: either use appliance or:
- Install CentOS Linux 8 minimal from
http://centos.mirror.garr.it/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-...
- dnf install
https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm
- dnf update (reboot if needed)
- dnf module enable -y javapackages-tools pki-deps 389-ds
- dnf install ovirt-engine
- engine-setup
For the nodes:
Either use oVirt Node ISO or:
- Install CentOS Linux 8 from
http://centos.mirror.garr.it/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-...
; select minimal installation
- dnf install
https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm
- dnf update (reboot if needed)
- Attach the host to engine and let it be deployed.
What’s new in oVirt 4.4.0 Beta?
-
Hypervisors based on CentOS Linux 8 (rebuilt from award winning RHEL8),
for both oVirt Node and standalone CentOS Linux hosts
-
Easier network management and configuration flexibility with
NetworkManager
-
VMs based on a more modern Q35 chipset with legacy seabios and UEFI
firmware
-
Support for direct passthrough of local host disks to VMs
-
Live migration improvements for High Performance guests.
-
New Windows Guest tools installer based on WiX framework now moved to
VirtioWin project
-
Dropped support for cluster level prior to 4.2
-
Dropped SDK3 support
-
4K disks support
-
Exporting a VM to a data domain
-
Editing of floating disks
-
Integrating ansible-runner into engine, which allows a more detailed
monitoring of playbooks executed from engine
-
Adding/reinstalling hosts are now completely based on Ansible
-
The OpenStack Neutron Agent cannot be configured by oVirt anymore, it
should be configured by TripleO instead
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 8.1 or newer
* CentOS Linux (or similar) 8.1 or newer
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 8.1 or newer
* CentOS Linux (or similar) 8.1 or newer
* oVirt Node 4.4 based on CentOS Linux 8.1 (available for x86_64 only)
See the release notes [1] for installation instructions and a list of new
features and bugs fixed.
If you manage more than one oVirt instance, OKD or RDO we also recommend to
try ManageIQ <http://manageiq.org/>.
In such a case, please be sure to take the qc2 image and not the ova image.
Notes:
- oVirt Appliance is already available for CentOS Linux 8
- oVirt Node NG is already available for CentOS Linux 8
Additional Resources:
* Read more about the oVirt 4.4.0 release highlights:
http://www.ovirt.org/release/4.4.0/
* Get more oVirt project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.4.0/
[2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*
<https://www.redhat.com/en/summit?sc_cid=7013a000002D2QxAAK>*
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*
2 years, 11 months
Re: oVirt Storage quota problems
by Vrgotic, Marko
Dear oVirt,
Is there anyone willing and able to assist me in troubleshooting what seems to be multiple issues regarding quotas:
1. UI Exception each time I add Consumer to Quota.
2. User getting warning of not being authorized to ‘ConsumeQuota’ in order to AddVM or AddVMFromTemplate, even though User is in Consumers list.
3. Fishy quota usage percentage.
I see this strange behavior on all my platforms, so it’s starting to look as a bug to me.
Happy to provide any relevant logs or do tests needed.
Kindly awaiting your reply.
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
ActiveVideo
From: "Staniforth, Paul" <P.Staniforth(a)leedsbeckett.ac.uk>
Date: Friday, 13 March 2020 at 18:19
To: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com>, "users(a)ovirt.org" <users(a)ovirt.org>
Cc: Darko Stojchev <D.Stojchev(a)activevideo.com>
Subject: Re: oVirt Storage quota problems
Sorry haven't used quota in a while, one thought is there a template quota getting used?
Regards,
Paul S.
________________________________
From: Vrgotic, Marko <M.Vrgotic(a)activevideo.com>
Sent: 13 March 2020 15:15
To: Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users(a)ovirt.org <users(a)ovirt.org>
Cc: Stojchev, Darko <D.Stojchev(a)activevideo.com>
Subject: Re: oVirt Storage quota problems
Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe.
Hey Paul,
I did the check for another user that has same issue and what I found out its bit strange:
* DC1 / infrastructure1 – quota storage_inf1 – user is added in Consumers list - gets “2020-03-13 08:26:18,057Z WARN [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (default task-3804) [a6ba871a-dc42-4d4b-a01a-a72cd5a45959] Validation of action 'AddVmFromTemplate' failed for user azabaleta@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA”
* DC1 / development2 – quota storage_dev2 – user is added in Consumers list – works
The following logs confirm that:
* Fails against infrastructure1 :
“2020-03-13 08:26:18,057Z INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (default task-3804) [a6ba871a-dc42-4d4b-a01a-a72cd5a45959] No permission found for user '699687a1-da37-4b4
0-a86d-dc744208302d' or one of the groups he is member of, when running action 'AddVmFromTemplate', Required permissions are: Action type: 'USER' Action group: 'CONSUME_QUOTA' Object type: '
Quota' Object ID: '254f2582-839d-11e9-aaa2-00163e4f2a6d'.
2020-03-13 08:26:18,057Z WARN [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (default task-3804) [a6ba871a-dc42-4d4b-a01a-a72cd5a45959] Validation of action 'AddVmFromTemplate' failed for user azabaleta@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-13 08:27:03,426Z INFO [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (default task-3813) [1e11522e-b25a-4972-8ff8-1d9a08bd57ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[azabaleta-runner=VM_NAME]', sharedLocks='[a80c1f1f-9cd3-4e8c-bfe9-670aa36d2aff=DISK, 9c710118-bb68-45aa-bafd-0e90cb07b9cf=TEMPLATE]'}'
* Works against development2:
2020-03-13 08:27:04,323Z INFO [org.ovirt.engine.core.bll.AddGraphicsDeviceCommand] (default task-3813) [2cd83827] Running command: AddGraphicsDeviceCommand internal: true. Entities affected : ID: d75c0d37-4640-49b0-afd1-7db27541f4d4 Type: VMAction group EDIT_VM_PROPERTIES with role type USER
2020-03-13 08:27:04,379Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3813) [2cd83827] EVENT_ID: USER_ADD_VM_STARTED(37), VM azabaleta-runner creation was initiated by azabaleta@ictv.com(a)ictv.com-authz.
2020-03-13 08:27:04,777Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1e11522e-b25a-4972-8ff8-1d9a08bd57ca] Command 'CreateCloneOfTemplate' (id: '806f2e38-fc42-45c3-87ab-20bec50c226c') waiting on child command id: 'd3826f75-1095-4b5f-98c3-973f4683616b' type:'CopyImageGroupWithData' to complete
2020-03-13 08:27:04,777Z INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1e11522e-b25a-4972-8ff8-1d9a08bd57ca] Command 'CopyImageGroupWithData' (id: 'd3826f75-1095-4b5f-98c3-973f4683616b') waiting on child command id: 'eb9c3360-00a8-4539-bda4-89d70d22da68' type:'CreateVolumeContainer' to complete
2020-03-13 08:27:05,003Z INFO [org.ovirt.engine.core.bll.network.vm.ReorderVmNicsCommand] (default task-3813) [e9f67a0f-aaf8-45a6-bdae-d501e757a3b6] Running command: ReorderVmNicsCommand internal: false. Entities affected : ID: d75c0d37-4640-49b0-afd1-7db27541f4d4 Type: VMAction group CREATE_VM with role type USER
Please assist as it currently seems the Quota permissions are not working as expected. The Quotas storage_inf1 and storage_dev2 are configured in exact same way, expect that they are for different DataCenters.
From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com>
Date: Thursday, 12 March 2020 at 20:19
To: "Staniforth, Paul" <P.Staniforth(a)leedsbeckett.ac.uk>
Cc: "users(a)ovirt.org" <users(a)ovirt.org>, Darko Stojchev <D.Stojchev(a)activevideo.com>
Subject: Re: oVirt Storage quota problems
Hey Paul,
Thank you. I believe I already added him via Consumers tab, but it does not hurt to doublecheck.
Sent from my iPhone
On 12 Mar 2020, at 17:53, Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk> wrote:
Hello Marko,
if you select quota in the administration menu, then select the quota it is using you will see the consumers option and there it will allow you to add users or groups.
https://www.ovirt.org/documentation/admin-guide/chap-Quotas_and_Service_L...<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...>
Quotas and Service Level Agreement Policy | oVirt<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...>
Chapter 16: Quotas and Service Level Agreement Policy Introduction to Quota. Quota is a resource limitation tool provided with oVirt. Quota may be thought of as a layer of limitations on top of the layer of limitations set by User Permissions.
www.ovirt.org
Regards,
Paul S.
________________________________
From: Vrgotic, Marko <M.Vrgotic(a)activevideo.com>
Sent: 12 March 2020 15:46
To: users(a)ovirt.org <users(a)ovirt.org>
Cc: Stojchev, Darko <D.Stojchev(a)activevideo.com>
Subject: [ovirt-users] Re: oVirt Storage quota problems
Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe.
I fond the following:
“2020-03-12 14:03:05,260Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [b9635e29-4b8a-44c4-9611-e5af362c783c] No permission found for user '7f906bbf-d194-425f-b313-08a777b764ab' or one of the groups he is member of, when running action 'AddVm', Required permissions are: Action type: 'USER' Action group: 'CONSUME_QUOTA' Object type: 'Quota' Object ID: '47b5e9f4-67a4-4b78-abfd-175e3d9de8da'.
2020-03-12 14:03:05,260Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [b9635e29-4b8a-44c4-9611-e5af362c783c] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA”
Does anyone know which permission setting that is, or under which section belongs, as I cannot found clear check for it under Roles definition?
Kindly awaiting your reply.
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
ActiveVideo
From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com>
Date: Thursday, 12 March 2020 at 16:28
To: "users(a)ovirt.org" <users(a)ovirt.org>
Cc: Darko Stojchev <D.Stojchev(a)activevideo.com>
Subject: Re: oVirt Storage quota problems
Addiitonally, WARN from log file for one of the Users:
2020-03-12 14:03:05,218Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3607) [2bb50857-d018-48aa-b4ca-ed6fb5cd76bf] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:05,236Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3583) [4ef095e5-59fa-4c73-9035-c5251372066d] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:05,238Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3608) [a571bd5d-98af-4a92-bf74-40681c208d32] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:05,240Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3599) [a77740ff-e725-4632-a189-cac9979238df] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:05,260Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [b9635e29-4b8a-44c4-9611-e5af362c783c] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:30,584Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3599) [90da1953-a2d6-45bb-8a5b-8f8e9a1b7e2c] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:30,597Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [fb938e77-1b95-4405-bebc-b0dd89e99ff0] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:30,600Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3608) [d3c4cd69-134d-49c7-a977-c27d79ce2e24] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:43,460Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3599) [99b158c6-c476-43e4-838f-45cadf73c7e9] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:43,462Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3608) [300289be-d55d-468e-aeda-d845180621e2] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:43,467Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [a6d267af-d14e-44d5-9573-de33a4d45586] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:45,101Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [8efba949-8f58-427b-9add-ead7b54e69d5] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:55,071Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3613) [3444440d-5c3a-4ba7-9a6b-4de6ba71a2cd] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
2020-03-12 14:03:55,093Z WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-3608) [34e8e6f9-7a5d-4d86-92bc-a595a3ae7688] Validation of action 'AddVm' failed for user esuilen@ictv.com(a)ictv.com-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,USER_NOT_AUTHORIZED_TO_CONSUME_QUOTA
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
ActiveVideo
From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com>
Date: Thursday, 12 March 2020 at 16:09
To: "users(a)ovirt.org" <users(a)ovirt.org>
Cc: Darko Stojchev <D.Stojchev(a)activevideo.com>
Subject: oVirt Storage quota problems
Dear oVirt,
I am experiencing a quota related problem regarding two things:
1. Quota usage and exception trigger
2. User assigned as quota consumer
First issue:
I have defined quota for storage ( qstorage ), setting all following items:
a. Defining quota for storage, with threshold and grace
b. Setting quota mode to enforced in datacenter
c. Added Users as consumers of the Quota
d. Updated quota from Default to qstorage for Templates and related images
e. Updated quota from Default to qstorage from all existing VMs and the related Disks
Problem:
* the percentage of the Storage Consumption is jumping up and down, between 3% and 97%, and at certain point goes back to Unlimited
* when checking under Storage tab inside quota name, the value “Used MB out of Total MB” is correct, however, when attempting to exceed the threshold, I am not getting a warning, even though I am consumer of a quota
Second issue:
User is added to qstorage quota as consumer. Still, when he tries to launch the VMs, Waring is displayed:
Cannot add VM. The user is not a consumer of the Quota assigned to the resource.
Have I missed something – could it be a permission issue, not a consumer issue.
All my users are utilizing Administration Portal with limited set of permissions. They all have same Role assigned, but only few of them are having the problem I described as Second problem.
Please assist.
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.acti...>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com>
Date: Thursday, 27 February 2020 at 10:19
To: "users(a)ovirt.org" <users(a)ovirt.org>
Cc: Darko Stojchev <D.Stojchev(a)activevideo.com>
Subject: oVirt Storage quota questions
Dear oVirt,
My platform is running SHE oVirt version 4.3.8.
Why is my StorageQuota consumption still showing “ 0 out of 3700” ? Did I forget something?
<image001.png>
I have setup storage quota “StorageQuota” for one of domains.
It has 4400GB actual storage.
• Quota is set to 3700GB.
o Treshold for Quota is 85% and Grace is set to 110%.
• Datacenter Quota is set to Enforced
• The template image has been changed from Default to StorageQuota
• I see that Vms created from Template are automatically assigned StorageQuota
To present the following situation in screenshots:
• Actual Quota:
<image002.png>
• VMs (more than on the screenshot) which have the StorageQuota assigned:
<image003.png>
• Templates which have the StorageQuota assigned
<image004.png>
Kindly awaiting your reply.
Additionally – is there an easy&safe way to update all running VMs quota?
-----
kind regards/met vriendelijke groeten
Marko Vrgotic
Sr. System Engineer @ System Administration
ActiveVideo
o: +31 (35) 6774131
e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com>
w: www.activevideo.com<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.acti...>
ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.
To view the terms under which this email is distributed, please go to:-
http://leedsbeckett.ac.uk/disclaimer/email/
To view the terms under which this email is distributed, please go to:-
http://leedsbeckett.ac.uk/disclaimer/email/
2 years, 11 months
Problem with connecting the Storage Domain (Host host2 cannot access the Storage Domain(s) <UNKNOWN>)
by Patrick Lomakin
Hello, everyone! I need specialist help, because I'm already desperate. My company has four hosts that are connected to the vault. Each host has its own IP to access the storage, which means host 1 has an ip 10.42.0.10 and 10.42.1.10 -> host 2 has an ip 10.42.0.20 and 10.42.0.20 respectively. Host 1 cannot ping the address 10.42.0.20. Hardware I tried to explain in more detail.
Host 1 has ovirt node 4.3.9 installed and hosted-engine deployed.
When trying to add host 2 to a cluster it is installed, but not activated. There is an error in ovirt manager - "Host host2 cannot access the Storage Domain(s) <UNKNOWN>" and host 2 goes to "Not operational" status. On host 2, it writes "connect to 10.42.1.10:3260 failed (No route to host)" in the logs and repeats indefinitely. I manually connected host 2 to the storage using iscsiadm to ip 10.42.0.20. But the error is not missing(. At the same time, when the host tries to activate it, I can run virtual machines on it until the host shows an error message. VMs that have been run on host 2 continue to run even when the host has "Not operational" status.
I assume that when adding host 2 to a cluster, ovirt tries to connect it to the same repository host 1 is connected to from ip 10.42.1.10. There may be a way to get ovirt to connect to another ip address instead of the ip domain address for the first host. I'm attaching logs:
----> /var/log/ovirt-engine/engine.log
2020-03-31 09:13:03,866+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghan dling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [7fa128f4] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host host2.school34.local c annot access the Storage Domain(s) <UNKNOWN> attached to the Data Center DataCenter. Setting Host state to Non-Operational.
2020-03-31 10:40:04,883+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [7a48ebb7] START, ConnectStorageServerVDSCommand(HostName = host2.school34.local, StorageServerConnectionManagementVDSParameters:{hostId='d82c3a76-e417-4fe4-8b08-a29414e3a9c1', storagePoolId='6052cc0a-71b9-11ea-ba5a-00163e10c7e7', storageType='ISCSI', connectionList='[StorageServerConnections:{id='c8a05dc2-f8a2-4354-96ed-907762c29761', connection='10.42.0.10', iqn='iqn.2002-09.com.lenovo:01.array.00c0ff3bfcb0', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='0ec6f34e-01c8-4ecc-9bd4-7e2a250d589d', connection='10.42.1.10', iqn='iqn.2002-09.com.lenovo:01.array.00c0ff3bfcb0', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnF
ailure='true'}), log id: 2c1a22b5
2020-03-31 10:43:05,061+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [7a48ebb7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host2.school34.local command ConnectStorageServerVDS failed: Message timeout which can be caused by communication issues
----> vdsm.log
2020-03-31 09:34:07,264+0300 ERROR (jsonrpc/5) [storage.HSM] Could not connect to storageServer (hsm:2420)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2417, in connectStorageServer
conObj.connect()
File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 488, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 217, in addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 337, in node_login
raise IscsiNodeError(rc, out, err)
IscsiNodeError: (8, ['Logging in to [iface: default, target: iqn.2002-09.com.lenovo:01.array.00c0ff3bfcb0, portal: 10.42.1.10,3260] (multiple)'], ['iscsiadm: Could not login to [iface: default, target: iqn.2002-09.com.lenovo:01.array.00c0ff3bfcb0, portal: 10.42.1.10,3260].', 'iscsiadm: initiator reported error (8 - connection timed out)', 'iscsiadm: Could not log into all portals'])
2020-03-31 09:36:01,583+0300 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/0 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'port': u'3260', u'connection': u'10.42.0.10', u'iqn': u'iqn.2002-09.com.lenovo:01.array.00c0ff3bfcb0', u'user': u'', u'tpgt': u'2', u'ipv6_enabled': u'false', u'password': '********', u'id': u'c8a05dc2-f8a2-4354-96ed-907762c29761'}, {u'port': u'3260', u'connection': u'10.42.1.10', u'iqn': u'iqn.2002-09.com.lenovo:01.array.00c0ff3bfcb0', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', u'password': '********', u'id': u'0ec6f34e-01c8-4ecc-9bd4-7e2a250d589d'}], u'storagepoolID': u'6052cc0a-71b9-11ea-ba5a-00163e10c7e7', u'domainType': 3}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'64cc0385-3a11-474b-98f0-b0ecaa6c67c8'} at 0x7fe1ac1ff510> timeout=60, duration=60.00 at 0x7fe1ac1ffb10> task#=316 at 0x7fe1f0041ad0>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 260, in run
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task
task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__
self._callable()
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__
self._handler(self._ctx, self._req)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest
response = self._handle_request(req, ctx)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request
res = method(**params)
File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod
result = fn(*methodArgs)
File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer
connectionParams)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper
result = ctask.prepare(func, *args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper
return m(self, *a, **kw)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare
result = self._run(func, *args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
return fn(*args, **kargs)
File: "<string>", line 2, in connectStorageServer
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2417, in connectStorageServer
conObj.connect()
File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 488, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 217, in addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File: "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 327, in node_login
portal, "-l"])
File: "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 122, in _runCmd
return misc.execCmd(cmd, printable=printCmd, sudo=True, sync=sync)
File: "/usr/lib/python2.7/site-packages/vdsm/common/commands.py", line 213, in execCmd
(out, err) = p.communicate(data)
File: "/usr/lib64/python2.7/site-packages/subprocess32.py", line 924, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File: "/usr/lib64/python2.7/site-packages/subprocess32.py", line 1706, in _communicate
orig_timeout)
File: "/usr/lib64/python2.7/site-packages/subprocess32.py", line 1779, in _communicate_with_poll
ready = poller.poll(self._remaining_time(endtime)) (executor:363)
2 years, 11 months
Failing to redeploy self hosted engine
by Maton, Brett
I keep running into this error when I try to (re)deploy self-hosted engine.
# ovirt-hosted-engine-cleanup
# hosted-engine --deploy
...
...
[ INFO ] TASK [ovirt.hosted_engine_setup : Fail with error description]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The
host has been set in non_operational status, deployment errors: code
9000: Failed to verify Power Management configuration for Host
physhost01.example.com, fix accordingly and re-deploy."}
I shut down all of the VM's and detached the storage before cleaning up and
trying to re-deploy the hosted engine, first time I've run into this
particular problem.
Any help appreciated
Brett
2 years, 11 months
Artwork: 4.4 GA banners
by Sandro Bonazzola
Hi,
in preparation of oVirt 4.4 GA it would be nice to have some graphics we
can use for launching oVirt 4.4 GA on social media and oVirt website.
If you don't have coding skills but you have marketing or design skills
this is a good opportunity to contribute back to the project.
Looking forward to your designs!
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*
<https://www.redhat.com/en/summit?sc_cid=7013a000002D2QxAAK>*
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*
2 years, 11 months
NFS permissions error on ISODomain file with correct permissions
by Shareef Jalloq
Hi,
I asked this question in another thread but it seems to have been lost in
the noise so I'm reposting with a more descriptive subject.
I'm trying to start a Windows VM and use the virtio-win VFD floppy to get
the drivers but the VM startup fails due to a permissions issue detailed
below. The permissions look fine to me so why can't the VFD be read?
Shareef.
I found a permissions issue in the engine.log:
2020-03-25 21:28:41,662Z ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-14) [] EVENT_ID: VM_DOWN_ERROR(119), VM win-2019 is
down with error. Exit message: internal error: qemu unexpectedly closed the
monitor: 2020-03-25T21:28:40.324426Z qemu-kvm: -drive
file=/rhev/data-center/mnt/nas-01.phoelex.com:_volume2_isostore/41cebb4b-c164-4956-8f44-6426170cd9f5/images/11111111-1111-1111-1111-111111111111/virtio-win_servers_amd64.vfd,format=raw,if=none,id=drive-ua-0b9c28b5-f75c-4575-ad85-b5b836f67d61,readonly=on:
Could not open '/rhev/data-center/mnt/nas-01.phoelex.com:_volume2_isostore/41cebb4b-c164-4956-8f44-6426170cd9f5/images/11111111-1111-1111-1111-111111111111/virtio-win_servers_amd64.vfd':
Permission denied.
But when I look at that path on the node in question, every folder and the
final file have the correct vdsm:kvm permissions:
[root@ovirt-node-01 ~]# ll /rhev/data-center/mnt/nas-01.phoelex.com:
_volume2_isostore/41cebb4b-c164-4956-8f44-6426170cd9f5/images/11111111-1111-1111-1111-111111111111/virtio-win_servers_amd64.vfd
-rwxrwxrwx. 1 vdsm kvm 2949120 Mar 25 21:24
/rhev/data-center/mnt/nas-01.phoelex.com:
_volume2_isostore/41cebb4b-c164-4956-8f44-6426170cd9f5/images/11111111-1111-1111-1111-111111111111/virtio-win_servers_amd64.vfd
The files were uploaded to the ISO domain using:
engine-iso-uploader --iso-domain=iso_storage upload virtio-win.iso
virtio-win_servers_amd64.vfd
2 years, 11 months
Re: Windows VirtIO drivers
by eevans＠digitaldatatechs.com
Ok. Personally, I like having an ISO repository, but I like the fact that's
it will be optional.
Thanks.
Eric Evans
Digital Data Services LLC.
304.660.9080
-----Original Message-----
From: Robert Webb <rwebb(a)ropeguru.com>
Sent: Wednesday, March 25, 2020 4:26 PM
To: eevans(a)digitaldatatechs.com; 'Shareef Jalloq' <shareef(a)jalloq.co.uk>;
users(a)ovirt.org
Subject: Re: [ovirt-users] Re: Windows VirtIO drivers
It does work that way.
I found that out as I was testing oVirt and had not created a separate ISO
Domain. I believe it was Strahil who pointed me in the right direction.
So if one does not have an ISO Domain, it is no longer required. Along with
the fact that ISO Domains are or in the process of being deprecated.
________________________________________
From: eevans(a)digitaldatatechs.com <eevans(a)digitaldatatechs.com>
Sent: Wednesday, March 25, 2020 3:13 PM
To: Robert Webb; 'Shareef Jalloq'; users(a)ovirt.org
Subject: RE: [ovirt-users] Re: Windows VirtIO drivers
That may be true, but in the ISO domain, when you open virt viewer you can
change the cd very easily...maybe it works that way as well..
Eric Evans
Digital Data Services LLC.
304.660.9080
-----Original Message-----
From: Robert Webb <rwebb(a)ropeguru.com>
Sent: Wednesday, March 25, 2020 2:35 PM
To: eevans(a)digitaldatatechs.com; 'Shareef Jalloq' <shareef(a)jalloq.co.uk>;
users(a)ovirt.org
Subject: [ovirt-users] Re: Windows VirtIO drivers
Don't think you have to use the ISO Domain any longer.
You can upload to a Data Domain and when you highlight the VM in the
management GUI, select the three dots in the top left for extra options and
there is a change cd option. That option will allow for attaching an ISO
from a Data Domain.
That is what I recall when I was using oVirt a month or so ago.
________________________________________
From: eevans(a)digitaldatatechs.com <eevans(a)digitaldatatechs.com>
Sent: Wednesday, March 25, 2020 2:28 PM
To: 'Shareef Jalloq'; users(a)ovirt.org
Subject: [ovirt-users] Re: Windows VirtIO drivers
You have to copy the iso and vfd files to the ISO domain to make them
available to the vm's that need drivers.
engine-iso-uploader options list
# engine-iso-uploader options upload file file file Documentation is found
here: https://www.ovirt.org/documentation/admin-guide/chap-Utilities.html
Eric Evans
Digital Data Services LLC.
304.660.9080
[cid:image001.jpg@01D602B1.9FEA2EC0]
From: Shareef Jalloq <shareef(a)jalloq.co.uk>
Sent: Wednesday, March 25, 2020 1:51 PM
To: users(a)ovirt.org
Subject: [ovirt-users] Windows VirtIO drivers
Hi,
it seems the online documentation regarding the windows installation steps
is well out of date. Where is there any current documentation on where to
get the VirtIO drivers for a Windows installation?
From a bit of Googling, it seems that I need to 'yum install virtio-win' on
the engine VM and then copy the relevant .iso/.vfd to the ISO domain. Is
that correct?
Where is the documentation maintained and how do I open a bug on it?
Thanks, Shareef.
2 years, 12 months
