January 2020 - Users - oVirt List Archives

Hyperconverged setup - storage architecture - scaling
by Leo David 10 Jan '20

10 Jan '20

Hello Everyone, Reading through the document: "Red Hat Hyperconverged Infrastructure for Virtualization 1.5 Automating RHHI for Virtualization deployment" Regarding storage scaling, i see the following statements: *2.7. SCALINGRed Hat Hyperconverged Infrastructure for Virtualization is supported for one node, and for clusters of 3, 6, 9, and 12 nodes.The initial deployment is either 1 or 3 nodes.There are two supported methods of horizontally scaling Red Hat Hyperconverged Infrastructure for Virtualization:* *1 Add new hyperconverged nodes to the cluster, in sets of three, up to the maximum of 12 hyperconverged nodes.* *2 Create new Gluster volumes using new disks on existing hyperconverged nodes.You cannot create a volume that spans more than 3 nodes, or expand an existing volume so that it spans across more than 3 nodes at a time* *2.9.1. Prerequisites for geo-replicationBe aware of the following requirements and limitations when configuring geo-replication:One geo-replicated volume onlyRed Hat Hyperconverged Infrastructure for Virtualization (RHHI for Virtualization) supports only one geo-replicated volume. Red Hat recommends backing up the volume that stores the data of your virtual machines, as this is usually contains the most valuable data.* ------ Also in oVirtEngine UI, when I add a brick to an existing volume i get the following warning: *"Expanding gluster volume in a hyper-converged setup is not recommended as it could lead to degraded performance. To expand storage for cluster, it is advised to add additional gluster volumes." * Those things are raising a couple of questions that maybe for some for you guys are easy to answer, but for me it creates a bit of confusion... I am also referring to RedHat product documentation, because I treat oVirt as production-ready as RHHI is. *1*. Is there any reason for not going to distributed-replicated volumes ( ie: spread one volume across 6,9, or 12 nodes ) ? - ie: is recomanded that in a 9 nodes scenario I should have 3 separated volumes, but how should I deal with the folowing question *2.* If only one geo-replicated volume can be configured, how should I deal with 2nd and 3rd volume replication for disaster recovery *3.* If the limit of hosts per datacenter is 250, then (in theory ) the recomended way in reaching this treshold would be to create 20 separated oVirt logical clusters with 12 nodes per each ( and datacenter managed from one ha-engine ) ? *4.* In present, I have the folowing one 9 nodes cluster , all hosts contributing with 2 disks each to a single replica 3 distributed replicated volume. They where added to the volume in the following order: node1 - disk1 node2 - disk1 ...... node9 - disk1 node1 - disk2 node2 - disk2 ...... node9 - disk2 At the moment, the volume is arbitrated, but I intend to go for full distributed replica 3. Is this a bad setup ? Why ? It oviously brakes the redhat recommended rules... Is there anyone so kind to discuss on these things ? Thank you very much ! Leo -- Best regards, Leo David -- Best regards, Leo David

3 5

Re: Ovirt OVN help needed
by Strahil 10 Jan '20

10 Jan '20

Hi Dominik, Thanks for your reply. On ovirt1 I got the following: [root@ovirt1 openvswitch]# less ovn-controller.log-20191216.gz 2019-12-15T01:49:02.988Z|00032|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2019-12-16T01:18:02.114Z|00033|vlog|INFO|closing log file ovn-controller.log-20191216.gz (END) Same is on the other node: [root@ovirt2 openvswitch]# less ovn-controller.log-20191216.gz 2019-12-15T01:26:03.477Z|00028|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2019-12-16T01:30:01.718Z|00029|vlog|INFO|closing log file ovn-controller.log-20191216.gz (END) The strange thing is that the geneve tunnels are there: [root@ovirt1 ~]# ovs-vsctl show c0e938f1-b5b5-4d5a-9cda-29dae2986f29 Bridge br-int fail_mode: secure Port "ovn-25cc77-0" Interface "ovn-25cc77-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.64"} Port "ovn-566849-0" Interface "ovn-566849-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.41"} Port br-int Interface br-int type: internal Port "vnet2" Interface "vnet2" ovs_version: "2.11.0" [root@ovirt1 ~]# ovs-vsctl list ports ovs-vsctl: unknown table "ports" [root@ovirt1 ~]# ovs-vsctl list port _uuid : fbf40569-925e-4430-a7c5-c78d58979bbc bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [3207c0cb-3000-40f2-a850-83548f76f090]lacp : [] mac : [] name : "vnet2" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 8947f82d-a089-429b-8843-71371314cb52 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [ec6a6688-e5d6-4346-ac47-ece1b8379440]lacp : [] mac : [] name : br-int other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 72d612be-853e-43e9-8f5c-ce66cef0bebe bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="5668499c-7dd0-41ee-bc5d-2e6ee9cd61c3(a)192.168.1.41"} fake_bridge : false interfaces : [a31574fe-515b-420b-859d-7f2ac729638f]lacp : [] mac : [] name : "ovn-566849-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 2043a15f-ec39-4cc3-b875-7be00423dd7a bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="25cc77b3-046f-45c5-af0c-ffb2f77d73f1(a)192.168.1.64"} fake_bridge : false interfaces : [f9a9e3ff-070e-4044-b601-7f7394dc295f]lacp : [] mac : [] name : "ovn-25cc77-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] [root@ovirt1 ~]# [root@ovirt2 ~]# ovs-vsctl show 3dbab138-6b90-44c5-af05-b8a944c9bf20 Bridge br-int fail_mode: secure Port "ovn-baa019-0" Interface "ovn-baa019-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.90"} Port br-int Interface br-int type: internal Port "vnet5" Interface "vnet5" Port "ovn-566849-0" Interface "ovn-566849-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.41"} ovs_version: "2.11.0" [root@ovirt2 ~]# ovs-vsctl list port _uuid : 151e1188-f07a-4750-a620-392a08e7e7fe bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="baa0199e-d1a4-484c-af13-a41bcad19dbc(a)192.168.1.90"} fake_bridge : false interfaces : [4d4bc12a-609a-4917-b839-d4f652acdc33]lacp : [] mac : [] name : "ovn-baa019-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 3a862f96-b3ec-46a9-bcf6-f385e5def410 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [777f2819-ca27-4890-8d2f-11349ca0d398]lacp : [] mac : [] name : br-int other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : a65109fa-f8b4-4670-8ae8-a2bd0bf6aba3 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="5668499c-7dd0-41ee-bc5d-2e6ee9cd61c3(a)192.168.1.41"} fake_bridge : false interfaces : [ed442077-f897-4e0b-97a1-a8051e9c3d56]lacp : [] mac : [] name : "ovn-566849-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : a1622e6f-fcd0-4a8a-b259-ca4d0ccf1cd2 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [ca368654-54f3-49d0-a71c-8894426df6bf]lacp : [] mac : [] name : "vnet5" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] [root@ovirt2 ~]# Best Regards, Strahil Nikolov On Dec 16, 2019 23:28, Dominik Holler <dholler(a)redhat.com> wrote: > > > > On Sat, Dec 14, 2019 at 11:36 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Hi Dominik, >> >> yes I was looking for those settings. >> >> I have added again the external provider , but I guess the mess is even bigger as I made some stupid decisions (like removing 2 port groups :) without knowing what I'm doing) . >> Sadly I can't remove all packages on the engine and hosts and reinstall them from scratch. >> >> Pip fails to install the openstacksdk (centOS7 is not great for such tasks) on the engine and my lack of knowledge in OVN makes it even more difficult. >> >> So the symptoms are that 2 machines can communicate with each other only if they are on the same host ,while on separate - no communications is happening. >> > > This indicates that the tunnels between the hosts are not created. > Can you please check the /var/log/openvswitch/ovn-controller.log on both hosts for errors and warnings, or share parts of the files here? > If this does not point us to a problem, ovn has to be reconfigured. If possible, most easy way to do this would be to ensure that > ovirt-provider-ovn is the default network provider of the cluster of the hosts, put one host after another in maintance mode and reinstall. > > >> >> How I created the network via UI: >> >> 1. Networks - new >> 2. Fill in the name >> 3. Create on external provider >> 4. Network Port security -> disabled (even undefined does not work) >> 5.Connect to physical network -> ovirtmgmt >> >> >> I would be happy to learn more about OVN and thus I would like to make it work. >> >> Here is some info from the engine: >> >> [root@engine ~]# ovn-nbctl show >> switch 1288ed26-471c-4bc2-8a7d-4531f306f44c (ovirt-pxelan-2a88b2e0-d04b-4196-ad50-074501e4ed08) >> port c1eba112-5eed-4c04-b25c-d3dcfb934546 >> addresses: ["56:6f:5a:65:00:06"] >> port 8b52ab60-f474-4d51-b258-cb2e0a53c34a >> type: localnet >> addresses: ["unknown"] >> port b2753040-881b-487a-92a1-9721da749be4 >> addresses: ["56:6f:5a:65:00:09"] >> [root@engine ~]# ovn-sbctl show >> Chassis "5668499c-7dd0-41ee-bc5d-2e6ee9cd61c3" >> hostname: "ovirt3.localdomain" >> Encap geneve >> ip: "192.168.1.41" >> options: {csum="true"} >> Chassis "baa0199e-d1a4-484c-af13-a41bcad19dbc" >> hostname: "ovirt1.localdomain" >> Encap geneve >> ip: "192.168.1.90" >> options: {csum="true"} >> Chassis "25cc77b3-046f-45c5-af0c-ffb2f77d73f1" >> hostname: "ovirt2.localdomain" >> Encap geneve >> ip: "192.168.1.64" >> options: {csum="true"} >> Port_Binding "b2753040-881b-487a-92a1-9721da749be4" >> Port_Binding &quo

4 8

is there any feature of load balancing for engine??
by yam yam 10 Jan '20

10 Jan '20

Hello, given massive oVirt environment, I think single engine looks too small to deal with all workloads. so, I want to make active-active engine cluster for distributing workloads. is it possible for an oVirt environment to be made up of multiple engines & DBs for load balancing?

3 4

Re: is there any feature of load balancing for engine??
by Strahil 10 Jan '20

10 Jan '20

It is not a good idea. If you think that the engine can't cope with the load, add a separate engine for the specific cluster(s). What is the count of Datacenters, Clusters, Hosts, VMs & Storage domains ? Best Regards, Strahil NikolovOn Jan 8, 2020 03:51, yam yam <hyunooudy(a)gmail.com> wrote: > > Hello, > > given massive oVirt environment, I think single engine looks too small to deal with all workloads. > so, I want to make active-active engine cluster for distributing workloads. > is it possible for an oVirt environment to be made up of multiple engines & DBs for load balancing? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XCO5DJS63YKC6…

2 1

Re: Setting up cockpit?
by Strahil 10 Jan '20

10 Jan '20

On Jan 9, 2020 18:58, m.skrzetuski(a)gmail.com wrote: > > Hello everyone, > > I'd like to get cockpit to work because currently when I click "Host Console" on a host I just get "connection refused". I checked and after the engine installation the cockpit service was not running. When I start it, it runs and answers on port 9090, however the SSL certificate is broken. > > - How do I auto enable cockpit on installation? systemctl enable --now cockpit.socket -H < destination server> > - How do I supply my own SSL certification to cockpit? Put them in /etc/cockpit/ws-certs.d/ Source:https://www.redhat.com/en/blog/linux-system-administration-managemen… > > Kind regards > Skrzetuski Best Regards, Strahil Nikolov

1 0

Re: After NAS crash, one VM will not start up, Cannot prepare illegal volume
by David Johnson 09 Jan '20

09 Jan '20

Thank you again. After updating legality to LEGAL, [root@mx-ovirt-host2 ~]# vdsm-client Volume getInfo storagepoolID=25cd9bfc-bab6-11e8-90f3-78acc0b47b4d storagedomainID=6e627364-5e0c-4250-ac95-7cd914d0175f imageID=4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6 volumeID=f8066c56-6db1-4605-8d7c-0739335d30b8 { "status": "OK", "lease": { "path": "/rhev/data-center/mnt/192.168.2.220: _mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease", "owners": [], "version": null, "offset": 0 }, "domain": "6e627364-5e0c-4250-ac95-7cd914d0175f", "capacity": "1503238553600", "voltype": "LEAF", "description": "", "parent": "a912e388-d80d-4f56-805b-ea5e2f35d741", "format": "COW", "generation": 0, "image": "4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6", "uuid": "f8066c56-6db1-4605-8d7c-0739335d30b8", "disktype": "DATA", "legality": "LEGAL", "mtime": "0", "apparentsize": "36440899584", "truesize": "16916186624", "type": "SPARSE", "children": [], "pool": "", "ctime": "1571669201" } Attempt to start the VM result are: Log excerpt: 2020-01-09 06:47:46,575-0600 INFO (vm/c5d0a42f) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6 to /var/run/vdsm/storage/6e627364-5e0c-4250-ac95-7cd914d0175f/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6 (fileSD:580) 2020-01-09 06:47:46,581-0600 INFO (vm/c5d0a42f) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', 'imgVolumesInfo': [{'domainID': '6e627364-5e0c-4250-ac95-7cd914d0175f', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/a912e388-d80d-4f56-805b-ea5e2f35d741', 'volumeID': u'a912e388-d80d-4f56-805b-ea5e2f35d741', 'leasePath': u'/rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/a912e388-d80d-4f56-805b-ea5e2f35d741.lease', 'imageID': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6'}, {'domainID': '6e627364-5e0c-4250-ac95-7cd914d0175f', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', 'volumeID': u'f8066c56-6db1-4605-8d7c-0739335d30b8', 'leasePath': u'/rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', 'imageID': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6'}]} from=internal, task_id=865d2ff4-4e63-44dc-b8f8-9d93cad9892f (api:52) 2020-01-09 06:47:46,582-0600 INFO (vm/c5d0a42f) [vds] prepared volume path: /rhev/data-center/mnt/192.168.2.220:_mnt_ovirt-freenas/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8 (clientIF:497) 2020-01-09 06:47:46,583-0600 INFO (vm/c5d0a42f) [vdsm.api] START prepareImage(sdUUID='ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', spUUID='25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', imgUUID='60077050-6f99-41db-b280-446f018b104b', leafUUID='a67eb40c-e0a1-49cc-9179-bebb263d6e9c', allowIllegal=False) from=internal, task_id=08830292-0f75-4c5b-a411-695894c66475 (api:46) 2020-01-09 06:47:46,632-0600 INFO (vm/c5d0a42f) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'a67eb40c-e0a1-49cc-9179-bebb263d6e9c',) from=internal, task_id=08830292-0f75-4c5b-a411-695894c66475 (api:50) 2020-01-09 06:47:46,632-0600 ERROR (vm/c5d0a42f) [storage.TaskManager.Task] (Task='08830292-0f75-4c5b-a411-695894c66475') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3187, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'a67eb40c-e0a1-49cc-9179-bebb263d6e9c',) 2020-01-09 06:47:46,633-0600 INFO (vm/c5d0a42f) [storage.TaskManager.Task] (Task='08830292-0f75-4c5b-a411-695894c66475') aborting: Task is aborted: "Cannot prepare illegal volume: (u'a67eb40c-e0a1-49cc-9179-bebb263d6e9c',)" - code 227 (task:1181) 2020-01-09 06:47:46,634-0600 ERROR (vm/c5d0a42f) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'a67eb40c-e0a1-49cc-9179-bebb263d6e9c',) (dispatcher:82) 2020-01-09 06:47:46,634-0600 ERROR (vm/c5d0a42f) [virt.vm] (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') The vm start process failed (vm:949) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 878, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2798, in _run self._devices = self._make_devices() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2639, in _make_devices disk_objs = self._perform_host_local_adjustment() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2712, in _perform_host_local_adjustment self._preparePathsForDrives(disk_params) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1023, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 417, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '3'}, 'serial': '60077050-6f99-41db-b280-446f018b104b', 'index': 1, 'iface': 'scsi', 'apparentsize': '16842752', 'specParams': {}, 'cache': 'writeback', 'imageID': '60077050-6f99-41db-b280-446f018b104b', 'truesize': '2840576', 'type': 'disk', 'domainID': 'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', 'reqsize': '0', 'format': 'cow', 'poolID': '25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', 'device': 'disk', 'path': '/rhev/data-center/25cd9bfc-bab6-11e8-90f3-78acc0b47b4d/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/60077050-6f99-41db-b280-446f018b104b/a67eb40c-e0a1-49cc-9179-bebb263d6e9c', 'propagateErrors': 'off', 'name': 'sdb', 'volumeID': 'a67eb40c-e0a1-49cc-9179-bebb263d6e9c', 'diskType': 'file', 'alias': 'ua-60077050-6f99-41db-b280-446f018b104b', 'discard': False} 2020-01-09 06:47:46,634-0600 INFO (vm/c5d0a42f) [virt.vm] (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') Changed state to Down: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '3'}, 'serial': '60077050-6f99-41db-b280-446f018b104b', 'index': 1, 'iface': 'scsi', 'apparentsize': '16842752', 'specParams': {}, 'cache': 'writeback', 'imageID': '60077050-6f99-41db-b280-446f018b104b', 'truesize': '2840576', 'type': 'disk', 'domainID': 'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', 'reqsize': '0', 'format': 'cow', 'poolID': '25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', 'device': 'disk', 'path': '/rhev/data-center/25cd9bfc-bab6-11e8-90f3-78acc0b47b4d/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/60077050-6f99-41db-b280-446f018b104b/a67eb40c-e0a1-49cc-9179-bebb263d6e9c', 'propagateErrors': 'off', 'name': 'sdb', 'volumeID': 'a67eb40c-e0a1-49cc-9179-bebb263d6e9c', 'diskType': 'file', 'alias': 'ua-60077050-6f99-41db-b280-446f018b104b', 'discard': False} (code=1) (vm:1689) 2 Regards, David Johnson Director of Development, Maxis Technology 844.696.2947 ext 702 (o) | 479.531.3590 (c) djohnson(a)maxistechnology.com [image: Maxis Techncology] <http://www.maxistechnology.com> www.maxistechnology.com *stay connected <http://www.linkedin.com/in/pojoguy>* On Thu, Jan 9, 2020 at 6:39 AM Benny Zlotnik <bzlotnik(a)redhat.com> wrote: > this `/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.meta` > > > You can see here[1] an example for updating volume metadata using > vdsm-client > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1409380#c3 > > On Thu, Jan 9, 2020 at 2:26 PM David Johnson <djohnson(a)maxistechnology.com> > wrote: > >> Thank you for the quick response. >> >> Where do I find that? >> >> Regards, >> David Johnson >> Director of Development, Maxis Technology >> 844.696.2947 ext 702 (o) | 479.531.3590 (c) >> djohnson(a)maxistechnology.com >> >> >> [image: Maxis Techncology] <http://www.maxistechnology.com> >> www.maxistechnology.com >> >> >> *stay connected <http://www.linkedin.com/in/pojoguy>* >> >> >> On Thu, Jan 9, 2020 at 6:24 AM Benny Zlotnik <bzlotnik(a)redhat.com> wrote: >> >>> Did you change the volume metadata to LEGAL on the storage as well? >>> >>> >>> On Thu, Jan 9, 2020 at 2:19 PM David Johnson < >>> djohnson(a)maxistechnology.com> wrote: >>> >>>> We had a drive in our NAS fail, but afterwards one of our VM's will not >>>> start. >>>> >>>> The boot drive on the VM is (so near as I can tell) the only drive >>>> affected. >>>> >>>> I confirmed that the disk images (active and snapshot) are both valid >>>> with qemu. >>>> >>>> I followed the instructions at >>>> https://www.canarytek.com/2017/07/02/Recover_oVirt_Illegal_Snapshots.html to >>>> identify the snapshot images that were marked "invalid" and marked them as >>>> valid. >>>> >>>> update images set imagestatus=1 where imagestatus=4; >>>> >>>> >>>> >>>> Log excerpt from attempt to start VM: >>>> 2020-01-09 02:18:44,908-0600 INFO (vm/c5d0a42f) [vdsm.api] START >>>> prepareImage(sdUUID='6e627364-5e0c-4250-ac95-7cd914d0175f', >>>> spUUID='25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', >>>> imgUUID='4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', >>>> leafUUID='f8066c56-6db1-4605-8d7c-0739335d30b8', allowIllegal=False) >>>> from=internal, task_id=26053225-6569-4b73-abdd-7d6c7e15d1e9 (api:46) >>>> 2020-01-09 02:18:44,931-0600 INFO (vm/c5d0a42f) [vdsm.api] FINISH >>>> prepareImage error=Cannot prepare illegal volume: >>>> (u'f8066c56-6db1-4605-8d7c-0739335d30b8',) from=internal, >>>> task_id=26053225-6569-4b73-abdd-7d6c7e15d1e9 (api:50) >>>> 2020-01-09 02:18:44,932-0600 ERROR (vm/c5d0a42f) >>>> [storage.TaskManager.Task] (Task='26053225-6569-4b73-abdd-7d6c7e15d1e9') >>>> Unexpected error (task:875) >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line >>>> 882, in _run >>>> return fn(*args, **kargs) >>>> File "<string>", line 2, in prepareImage >>>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, >>>> in method >>>> ret = func(*args, **kwargs) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line >>>> 3187, in prepareImage >>>> raise se.prepareIllegalVolumeError(volUUID) >>>> prepareIllegalVolumeError: Cannot prepare illegal volume: >>>> (u'f8066c56-6db1-4605-8d7c-0739335d30b8',) >>>> 2020-01-09 02:18:44,932-0600 INFO (vm/c5d0a42f) >>>> [storage.TaskManager.Task] (Task='26053225-6569-4b73-abdd-7d6c7e15d1e9') >>>> aborting: Task is aborted: "Cannot prepare illegal volume: >>>> (u'f8066c56-6db1-4605-8d7c-0739335d30b8',)" - code 227 (task:1181) >>>> 2020-01-09 02:18:44,933-0600 ERROR (vm/c5d0a42f) [storage.Dispatcher] >>>> FINISH prepareImage error=Cannot prepare illegal volume: >>>> (u'f8066c56-6db1-4605-8d7c-0739335d30b8',) (dispatcher:82) >>>> 2020-01-09 02:18:44,933-0600 ERROR (vm/c5d0a42f) [virt.vm] >>>> (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') The vm start process failed >>>> (vm:949) >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 878, in >>>> _startUnderlyingVm >>>> self._run() >>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2798, >>>> in _run >>>> self._devices = self._make_devices() >>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2639, >>>> in _make_devices >>>> disk_objs = self._perform_host_local_adjustment() >>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2712, >>>> in _perform_host_local_adjustment >>>> self._preparePathsForDrives(disk_params) >>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1023, >>>> in _preparePathsForDrives >>>> drive['path'] = self.cif.prepareVolumePath(drive, self.id) >>>> File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 417, >>>> in prepareVolumePath >>>> raise vm.VolumeError(drive) >>>> VolumeError: Bad volume specification {'address': {'bus': '0', >>>> 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'serial': >>>> '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'index': 0, 'iface': 'scsi', >>>> 'apparentsize': '36440899584', 'specParams': {}, 'cache': 'writeback', >>>> 'imageID': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'truesize': >>>> '16916186624', 'type': 'disk', 'domainID': >>>> '6e627364-5e0c-4250-ac95-7cd914d0175f', 'reqsize': '0', 'format': 'cow', >>>> 'poolID': '25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', 'device': 'disk', 'path': >>>> '/rhev/data-center/25cd9bfc-bab6-11e8-90f3-78acc0b47b4d/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': >>>> 'f8066c56-6db1-4605-8d7c-0739335d30b8', 'diskType': 'file', 'alias': >>>> 'ua-4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'discard': False} >>>> 2020-01-09 02:18:44,934-0600 INFO (vm/c5d0a42f) [virt.vm] >>>> (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') Changed state to Down: Bad >>>> volume specification {'address': {'bus': '0', 'controller': '0', 'type': >>>> 'drive', 'target': '0', 'unit': '0'}, 'serial': >>>> '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'index': 0, 'iface': 'scsi', >>>> 'apparentsize': '36440899584', 'specParams': {}, 'cache': 'writeback', >>>> 'imageID': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'truesize': >>>> '16916186624', 'type': 'disk', 'domainID': >>>> '6e627364-5e0c-4250-ac95-7cd914d0175f', 'reqsize': '0', 'format': 'cow', >>>> 'poolID': '25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', 'device': 'disk', 'path': >>>> '/rhev/data-center/25cd9bfc-bab6-11e8-90f3-78acc0b47b4d/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': >>>> 'f8066c56-6db1-4605-8d7c-0739335d30b8', 'diskType': 'file', 'alias': >>>> 'ua-4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'discard': False} (code=1) >>>> (vm:1689) >>>> 2020-01-09 02:18:44,963-0600 INFO (vm/c5d0a42f) [virt.vm] >>>> (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') Stopping connection >>>> (guestagent:442) >>>> 2020-01-09 02:18:44,978-0600 INFO (jsonrpc/4) [api.virt] START >>>> destroy(gracefulAttempts=1) from=::ffff:192.168.2.15,57858, >>>> flow_id=4b57f1a6, vmId=c5d0a42f-3b1e-43ee-a567-7844654011f5 (api:46) >>>> 2 >>>> >>>> >>>> Log excerpt from attempt to migrate the affected drive to another >>>> storage device: >>>> 2020-01-09 05:27:14,362-0600 INFO (jsonrpc/2) [storage.VolumeManifest] >>>> ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8 >>>> info is {'status': 'OK', 'domain': 'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', >>>> 'voltype': 'LEAF', 'description': 'None', 'parent': >>>> 'a912e388-d80d-4f56-805b-ea5e2f35d741', 'format': 'COW', 'generation': 0, >>>> 'image': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'ctime': '1578559013', >>>> 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': >>>> '219008', 'children': [], 'pool': '', 'capacity': '1503238553600', 'uuid': >>>> u'f8066c56-6db1-4605-8d7c-0739335d30b8', 'truesize': '26624', 'type': >>>> 'SPARSE', 'lease': {'path': u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', >>>> 'owners': [], 'version': None, 'offset': 0}} (volume:282) >>>> 2020-01-09 05:27:14,362-0600 INFO (jsonrpc/2) [vdsm.api] FINISH >>>> getVolumeInfo return={'info': {'status': 'OK', 'domain': >>>> 'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', 'voltype': 'LEAF', 'description': >>>> 'None', 'parent': 'a912e388-d80d-4f56-805b-ea5e2f35d741', 'format': 'COW', >>>> 'generation': 0, 'image': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'ctime': >>>> '1578559013', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', >>>> 'apparentsize': '219008', 'children': [], 'pool': '', 'capacity': >>>> '1503238553600', 'uuid': u'f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> 'truesize': '26624', 'type': 'SPARSE', 'lease': {'path': >>>> u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', >>>> 'owners': [], 'version': None, 'offset': 0}}} >>>> from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731, >>>> task_id=28e885f7-df18-4ea5-bb6e-eaee7385b519 (api:52) >>>> 2020-01-09 05:27:14,363-0600 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] >>>> RPC call Volume.getInfo succeeded in 0.04 seconds (__init__:573) >>>> 2020-01-09 05:27:14,394-0600 INFO (jsonrpc/6) [vdsm.api] START >>>> sdm_copy_data(job_id=u'9a47a14f-d398-4fe8-81fa-59cba7122aa0', >>>> source={u'img_id': u'4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', u'sd_id': >>>> u'6e627364-5e0c-4250-ac95-7cd914d0175f', u'endpoint_type': u'div', >>>> u'vol_id': u'f8066c56-6db1-4605-8d7c-0739335d30b8'}, >>>> destination={u'generation': 0, u'img_id': >>>> u'4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', u'sd_id': >>>> u'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', u'endpoint_type': u'div', >>>> u'vol_id': u'f8066c56-6db1-4605-8d7c-0739335d30b8'}) >>>> from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731, >>>> task_id=f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (api:46) >>>> 2020-01-09 05:27:14,395-0600 INFO (jsonrpc/6) [vdsm.api] FINISH >>>> sdm_copy_data return=None from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731, >>>> task_id=f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (api:52) >>>> 2020-01-09 05:27:14,395-0600 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] >>>> RPC call SDM.copy_data succeeded in 0.00 seconds (__init__:573) >>>> 2020-01-09 05:27:14,396-0600 INFO (tasks/4) >>>> [storage.ThreadPool.WorkerThread] START task >>>> f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (cmd=<bound method Task.commit of >>>> <vdsm.storage.task.Task instance at 0x7ff3a42613b0>>, args=None) >>>> (threadPool:208) >>>> 2020-01-09 05:27:14,396-0600 INFO (tasks/4) [root] Running job >>>> u'9a47a14f-d398-4fe8-81fa-59cba7122aa0'... (jobs:183) >>>> 2020-01-09 05:27:14,429-0600 INFO (jsonrpc/1) [api.host] START >>>> getJobs(job_type=u'storage', >>>> job_ids=[u'9a47a14f-d398-4fe8-81fa-59cba7122aa0']) >>>> from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:46) >>>> 2020-01-09 05:27:14,430-0600 INFO (jsonrpc/1) [api.host] FINISH >>>> getJobs return={'status': {'message': 'Done', 'code': 0}, 'jobs': >>>> {u'9a47a14f-d398-4fe8-81fa-59cba7122aa0': {'status': 'running', 'job_type': >>>> 'storage', 'id': u'9a47a14f-d398-4fe8-81fa-59cba7122aa0', 'description': >>>> 'copy_data'}}} from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:52) >>>> 2020-01-09 05:27:14,430-0600 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] >>>> RPC call Host.getJobs succeeded in 0.00 seconds (__init__:573) >>>> 2020-01-09 05:27:14,459-0600 INFO (tasks/4) [storage.SANLock] >>>> Acquiring Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', >>>> offset=0) for host id 2 (clusterlock:386) >>>> 2020-01-09 05:27:14,496-0600 INFO (tasks/4) [storage.SANLock] >>>> Successfully acquired Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', >>>> offset=0) for host id 2 (clusterlock:424) >>>> 2020-01-09 05:27:14,498-0600 INFO (tasks/4) [storage.VolumeManifest] >>>> Volume: preparing volume >>>> 6e627364-5e0c-4250-ac95-7cd914d0175f/f8066c56-6db1-4605-8d7c-0739335d30b8 >>>> (volume:569) >>>> 2020-01-09 05:27:14,502-0600 INFO (tasks/4) [storage.SANLock] >>>> Releasing Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', >>>> offset=0) (clusterlock:487) >>>> 2020-01-09 05:27:14,504-0600 INFO (tasks/4) [storage.SANLock] >>>> Successfully released Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', >>>> path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', >>>> offset=0) (clusterlock:496) >>>> 2020-01-09 05:27:14,504-0600 ERROR (tasks/4) [root] Job >>>> u'9a47a14f-d398-4fe8-81fa-59cba7122aa0' failed (jobs:221) >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/vdsm/jobs.py", line 157, in run >>>> self._run() >>>> File >>>> "/usr/lib/python2.7/site-packages/vdsm/storage/sdm/api/copy_data.py", line >>>> 61, in _run >>>> with self._source.prepare(), self._dest.prepare(): >>>> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >>>> return self.gen.next() >>>> File >>>> "/usr/lib/python2.7/site-packages/vdsm/storage/sdm/api/copy_data.py", line >>>> 172, in prepare >>>> self.volume.prepare(rw=self._writable, justme=False) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line >>>> 574, in prepare >>>> raise se.prepareIllegalVolumeError(self.volUUID) >>>> prepareIllegalVolumeError: Cannot prepare illegal volume: >>>> ('f8066c56-6db1-4605-8d7c-0739335d30b8',) >>>> 2020-01-09 05:27:14,505-0600 INFO (tasks/4) [root] Job >>>> u'9a47a14f-d398-4fe8-81fa-59cba7122aa0' will be deleted in 3600 seconds >>>> (jobs:249) >>>> 2020-01-09 05:27:14,506-0600 INFO (tasks/4) >>>> [storage.ThreadPool.WorkerThread] FINISH task >>>> f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (threadPool:210) >>>> 2020-01-09 05:27:16,459-0600 INFO (jsonrpc/4) [api.host] START >>>> getJobs(job_type=u'storage', >>>> job_ids=[u'9a47a14f-d398-4fe8-81fa-59cba7122aa0']) >>>> from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:46) >>>> 2020-01-09 05:27:16,460-0600 INFO (jsonrpc/4) [api.host] FINISH >>>> getJobs return={'status': {'message': 'Done', 'code': 0}, 'jobs': >>>> {u'9a47a14f-d398-4fe8-81fa-59cba7122aa0': {'status': 'failed', 'error': >>>> {'message': "Cannot prepare illegal volume: >>>> ('f8066c56-6db1-4605-8d7c-0739335d30b8',)", 'code': 227}, 'job_type': >>>> 'storage', 'id': u'9a47a14f-d398-4fe8-81fa-59cba7122aa0', 'description': >>>> 'copy_data'}}} from=::ffff:192.168.2.15,57858, >>>> flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:52) >>>> 2 >>>> >>>> Regards, >>>> David Johnson >>>> Director of Development, Maxis Technology >>>> 844.696.2947 ext 702 (o) | 479.531.3590 (c) >>>> djohnson(a)maxistechnology.com >>>> >>>> >>>> [image: Maxis Techncology] <http://www.maxistechnology.com> >>>> www.maxistechnology.com >>>> >>>> >>>> *stay connected <http://www.linkedin.com/in/pojoguy>* >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/KKLZKSPCESYVJ… >>>> >>>

1 2

AWX and error using ovirt as an inventory source
by Gianluca Cecchi 09 Jan '20

09 Jan '20

Hello, I have awx 9.0.1 and ansible 2.8.5 in container of a CentOS 7.7 server. I'm trying to use oVirt 4.3.6.7-1.el7 as a source of an inventory in awx but I get error when syncing Find at bottom below the error messages. I see that in recent past (around June this year) there were some problems, but they should be solved now, correct? There was also a problem in syncing when some powered off VMs were present in oVirt env, but I think this solved too, correct? Any way to replicate / test from command line of awx container? I try some things but in command line I always get error regarding oVirt inventory script requires ovirt-engine-sdk-python >= 4.0.0 that I think depends on not using correct command line and/or not setting needed env. Thanks in advance, Gianluca 2.536 INFO Updating inventory 4: MYDC_OVIRT 3.011 INFO Reading Ansible inventory source: /var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/plugins/inventory/ovirt4.py 3.013 INFO Using VIRTUAL_ENV: /var/lib/awx/venv/ansible 3.013 INFO Using PATH: /var/lib/awx/venv/ansible/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 3.013 INFO Using PYTHONPATH: /var/lib/awx/venv/ansible/lib/python3.6/site-packages: Traceback (most recent call last): File "/var/lib/awx/venv/awx/bin/awx-manage", line 11, in <module> load_entry_point('awx==9.0.1.0', 'console_scripts', 'awx-manage')() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/__init__.py", line 158, in manage execute_from_command_line(sys.argv) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line utility.execute() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, **cmd_options) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(*args, **options) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1153, in handle raise exc File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1043, in handle venv_path=venv_path, verbosity=self.verbosity).load() File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 214, in load return self.command_to_json(base_args + ['--list']) File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 197, in command_to_json self.method, proc.returncode, stdout, stderr)) RuntimeError: ansible-inventory failed (rc=1) with stdout: stderr: ansible-inventory 2.8.5 config file = /etc/ansible/ansible.cfg configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3.6/site-packages/ansible executable location = /usr/bin/ansible-inventory python version = 3.6.8 (default, Oct 7 2019, 17:58:22) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] Using /etc/ansible/ansible.cfg as config file [WARNING]: * Failed to parse /var/lib/awx/venv/awx/lib64/python3.6/site- packages/awx/plugins/inventory/ovirt4.py with script plugin: Inventory script (/var/lib/awx/venv/awx/lib64/python3.6/site- packages/awx/plugins/inventory/ovirt4.py) had an execution error: File "/usr/lib/python3.6/site-packages/ansible/inventory/manager.py", line 268, in parse_source plugin.parse(self._inventory, self._loader, source, cache=cache) File "/usr/lib/python3.6/site-packages/ansible/plugins/inventory/script.py", line 161, in parse raise AnsibleParserError(to_native(e)) [WARNING]: Unable to parse /var/lib/awx/venv/awx/lib64/python3.6/site- packages/awx/plugins/inventory/ovirt4.py as an inventory source ERROR! No inventory was parsed, please check your configuration and options.

3 23

After NAS crash, one VM will not start up, Cannot prepare illegal volume
by David Johnson 09 Jan '20

09 Jan '20

We had a drive in our NAS fail, but afterwards one of our VM's will not start. The boot drive on the VM is (so near as I can tell) the only drive affected. I confirmed that the disk images (active and snapshot) are both valid with qemu. I followed the instructions at https://www.canarytek.com/2017/07/02/Recover_oVirt_Illegal_Snapshots.html to identify the snapshot images that were marked "invalid" and marked them as valid. update images set imagestatus=1 where imagestatus=4; Log excerpt from attempt to start VM: 2020-01-09 02:18:44,908-0600 INFO (vm/c5d0a42f) [vdsm.api] START prepareImage(sdUUID='6e627364-5e0c-4250-ac95-7cd914d0175f', spUUID='25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', imgUUID='4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', leafUUID='f8066c56-6db1-4605-8d7c-0739335d30b8', allowIllegal=False) from=internal, task_id=26053225-6569-4b73-abdd-7d6c7e15d1e9 (api:46) 2020-01-09 02:18:44,931-0600 INFO (vm/c5d0a42f) [vdsm.api] FINISH prepareImage error=Cannot prepare illegal volume: (u'f8066c56-6db1-4605-8d7c-0739335d30b8',) from=internal, task_id=26053225-6569-4b73-abdd-7d6c7e15d1e9 (api:50) 2020-01-09 02:18:44,932-0600 ERROR (vm/c5d0a42f) [storage.TaskManager.Task] (Task='26053225-6569-4b73-abdd-7d6c7e15d1e9') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in prepareImage File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3187, in prepareImage raise se.prepareIllegalVolumeError(volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: (u'f8066c56-6db1-4605-8d7c-0739335d30b8',) 2020-01-09 02:18:44,932-0600 INFO (vm/c5d0a42f) [storage.TaskManager.Task] (Task='26053225-6569-4b73-abdd-7d6c7e15d1e9') aborting: Task is aborted: "Cannot prepare illegal volume: (u'f8066c56-6db1-4605-8d7c-0739335d30b8',)" - code 227 (task:1181) 2020-01-09 02:18:44,933-0600 ERROR (vm/c5d0a42f) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: (u'f8066c56-6db1-4605-8d7c-0739335d30b8',) (dispatcher:82) 2020-01-09 02:18:44,933-0600 ERROR (vm/c5d0a42f) [virt.vm] (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') The vm start process failed (vm:949) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 878, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2798, in _run self._devices = self._make_devices() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2639, in _make_devices disk_objs = self._perform_host_local_adjustment() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2712, in _perform_host_local_adjustment self._preparePathsForDrives(disk_params) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1023, in _preparePathsForDrives drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 417, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'serial': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'index': 0, 'iface': 'scsi', 'apparentsize': '36440899584', 'specParams': {}, 'cache': 'writeback', 'imageID': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'truesize': '16916186624', 'type': 'disk', 'domainID': '6e627364-5e0c-4250-ac95-7cd914d0175f', 'reqsize': '0', 'format': 'cow', 'poolID': '25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', 'device': 'disk', 'path': '/rhev/data-center/25cd9bfc-bab6-11e8-90f3-78acc0b47b4d/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': 'f8066c56-6db1-4605-8d7c-0739335d30b8', 'diskType': 'file', 'alias': 'ua-4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'discard': False} 2020-01-09 02:18:44,934-0600 INFO (vm/c5d0a42f) [virt.vm] (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') Changed state to Down: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'serial': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'index': 0, 'iface': 'scsi', 'apparentsize': '36440899584', 'specParams': {}, 'cache': 'writeback', 'imageID': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'truesize': '16916186624', 'type': 'disk', 'domainID': '6e627364-5e0c-4250-ac95-7cd914d0175f', 'reqsize': '0', 'format': 'cow', 'poolID': '25cd9bfc-bab6-11e8-90f3-78acc0b47b4d', 'device': 'disk', 'path': '/rhev/data-center/25cd9bfc-bab6-11e8-90f3-78acc0b47b4d/6e627364-5e0c-4250-ac95-7cd914d0175f/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8', 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': 'f8066c56-6db1-4605-8d7c-0739335d30b8', 'diskType': 'file', 'alias': 'ua-4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'discard': False} (code=1) (vm:1689) 2020-01-09 02:18:44,963-0600 INFO (vm/c5d0a42f) [virt.vm] (vmId='c5d0a42f-3b1e-43ee-a567-7844654011f5') Stopping connection (guestagent:442) 2020-01-09 02:18:44,978-0600 INFO (jsonrpc/4) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:192.168.2.15,57858, flow_id=4b57f1a6, vmId=c5d0a42f-3b1e-43ee-a567-7844654011f5 (api:46) 2 Log excerpt from attempt to migrate the affected drive to another storage device: 2020-01-09 05:27:14,362-0600 INFO (jsonrpc/2) [storage.VolumeManifest] ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8 info is {'status': 'OK', 'domain': 'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', 'voltype': 'LEAF', 'description': 'None', 'parent': 'a912e388-d80d-4f56-805b-ea5e2f35d741', 'format': 'COW', 'generation': 0, 'image': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'ctime': '1578559013', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '219008', 'children': [], 'pool': '', 'capacity': '1503238553600', 'uuid': u'f8066c56-6db1-4605-8d7c-0739335d30b8', 'truesize': '26624', 'type': 'SPARSE', 'lease': {'path': u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', 'owners': [], 'version': None, 'offset': 0}} (volume:282) 2020-01-09 05:27:14,362-0600 INFO (jsonrpc/2) [vdsm.api] FINISH getVolumeInfo return={'info': {'status': 'OK', 'domain': 'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', 'voltype': 'LEAF', 'description': 'None', 'parent': 'a912e388-d80d-4f56-805b-ea5e2f35d741', 'format': 'COW', 'generation': 0, 'image': '4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', 'ctime': '1578559013', 'disktype': 'DATA', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '219008', 'children': [], 'pool': '', 'capacity': '1503238553600', 'uuid': u'f8066c56-6db1-4605-8d7c-0739335d30b8', 'truesize': '26624', 'type': 'SPARSE', 'lease': {'path': u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', 'owners': [], 'version': None, 'offset': 0}}} from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731, task_id=28e885f7-df18-4ea5-bb6e-eaee7385b519 (api:52) 2020-01-09 05:27:14,363-0600 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Volume.getInfo succeeded in 0.04 seconds (__init__:573) 2020-01-09 05:27:14,394-0600 INFO (jsonrpc/6) [vdsm.api] START sdm_copy_data(job_id=u'9a47a14f-d398-4fe8-81fa-59cba7122aa0', source={u'img_id': u'4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', u'sd_id': u'6e627364-5e0c-4250-ac95-7cd914d0175f', u'endpoint_type': u'div', u'vol_id': u'f8066c56-6db1-4605-8d7c-0739335d30b8'}, destination={u'generation': 0, u'img_id': u'4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6', u'sd_id': u'ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7', u'endpoint_type': u'div', u'vol_id': u'f8066c56-6db1-4605-8d7c-0739335d30b8'}) from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731, task_id=f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (api:46) 2020-01-09 05:27:14,395-0600 INFO (jsonrpc/6) [vdsm.api] FINISH sdm_copy_data return=None from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731, task_id=f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (api:52) 2020-01-09 05:27:14,395-0600 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call SDM.copy_data succeeded in 0.00 seconds (__init__:573) 2020-01-09 05:27:14,396-0600 INFO (tasks/4) [storage.ThreadPool.WorkerThread] START task f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x7ff3a42613b0>>, args=None) (threadPool:208) 2020-01-09 05:27:14,396-0600 INFO (tasks/4) [root] Running job u'9a47a14f-d398-4fe8-81fa-59cba7122aa0'... (jobs:183) 2020-01-09 05:27:14,429-0600 INFO (jsonrpc/1) [api.host] START getJobs(job_type=u'storage', job_ids=[u'9a47a14f-d398-4fe8-81fa-59cba7122aa0']) from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:46) 2020-01-09 05:27:14,430-0600 INFO (jsonrpc/1) [api.host] FINISH getJobs return={'status': {'message': 'Done', 'code': 0}, 'jobs': {u'9a47a14f-d398-4fe8-81fa-59cba7122aa0': {'status': 'running', 'job_type': 'storage', 'id': u'9a47a14f-d398-4fe8-81fa-59cba7122aa0', 'description': 'copy_data'}}} from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:52) 2020-01-09 05:27:14,430-0600 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getJobs succeeded in 0.00 seconds (__init__:573) 2020-01-09 05:27:14,459-0600 INFO (tasks/4) [storage.SANLock] Acquiring Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', offset=0) for host id 2 (clusterlock:386) 2020-01-09 05:27:14,496-0600 INFO (tasks/4) [storage.SANLock] Successfully acquired Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', offset=0) for host id 2 (clusterlock:424) 2020-01-09 05:27:14,498-0600 INFO (tasks/4) [storage.VolumeManifest] Volume: preparing volume 6e627364-5e0c-4250-ac95-7cd914d0175f/f8066c56-6db1-4605-8d7c-0739335d30b8 (volume:569) 2020-01-09 05:27:14,502-0600 INFO (tasks/4) [storage.SANLock] Releasing Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', offset=0) (clusterlock:487) 2020-01-09 05:27:14,504-0600 INFO (tasks/4) [storage.SANLock] Successfully released Lease(name='f8066c56-6db1-4605-8d7c-0739335d30b8', path=u'/rhev/data-center/mnt/192.168.2.223:_mnt_ovirt-main-pool/ec6ccb14-03c2-49cc-9cc0-b1a87d582ed7/images/4081ce8f-1ce1-4ee1-aa43-69af2dfc5ab6/f8066c56-6db1-4605-8d7c-0739335d30b8.lease', offset=0) (clusterlock:496) 2020-01-09 05:27:14,504-0600 ERROR (tasks/4) [root] Job u'9a47a14f-d398-4fe8-81fa-59cba7122aa0' failed (jobs:221) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/jobs.py", line 157, in run self._run() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdm/api/copy_data.py", line 61, in _run with self._source.prepare(), self._dest.prepare(): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdm/api/copy_data.py", line 172, in prepare self.volume.prepare(rw=self._writable, justme=False) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 574, in prepare raise se.prepareIllegalVolumeError(self.volUUID) prepareIllegalVolumeError: Cannot prepare illegal volume: ('f8066c56-6db1-4605-8d7c-0739335d30b8',) 2020-01-09 05:27:14,505-0600 INFO (tasks/4) [root] Job u'9a47a14f-d398-4fe8-81fa-59cba7122aa0' will be deleted in 3600 seconds (jobs:249) 2020-01-09 05:27:14,506-0600 INFO (tasks/4) [storage.ThreadPool.WorkerThread] FINISH task f1504b2d-c6c2-4866-97ce-e0476b0ba9a2 (threadPool:210) 2020-01-09 05:27:16,459-0600 INFO (jsonrpc/4) [api.host] START getJobs(job_type=u'storage', job_ids=[u'9a47a14f-d398-4fe8-81fa-59cba7122aa0']) from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:46) 2020-01-09 05:27:16,460-0600 INFO (jsonrpc/4) [api.host] FINISH getJobs return={'status': {'message': 'Done', 'code': 0}, 'jobs': {u'9a47a14f-d398-4fe8-81fa-59cba7122aa0': {'status': 'failed', 'error': {'message': "Cannot prepare illegal volume: ('f8066c56-6db1-4605-8d7c-0739335d30b8',)", 'code': 227}, 'job_type': 'storage', 'id': u'9a47a14f-d398-4fe8-81fa-59cba7122aa0', 'description': 'copy_data'}}} from=::ffff:192.168.2.15,57858, flow_id=94ad3bf0-c5f5-4d45-af2a-49aa4eaa6731 (api:52) 2 Regards, David Johnson Director of Development, Maxis Technology 844.696.2947 ext 702 (o) | 479.531.3590 (c) djohnson(a)maxistechnology.com [image: Maxis Techncology] <http://www.maxistechnology.com> www.maxistechnology.com *stay connected <http://www.linkedin.com/in/pojoguy>*

2 3

Storage I/O problem.
by Christian Reiss 09 Jan '20

09 Jan '20

Ugh, After having rebooted the 3-way HCI Cluster everything came back online, all the gluster volumes are online, no split-brain info detected. Also they are mounted on all nodes. The Volumes, Domains and Disks are all marked green in the Engine. Launching a VM fails with "VM test01 has been paused due to storage I/O problem." A md5 sum of an uploaded ISO image from all three mounted cluster members yield the same md5, so does the source file on my PC. Creating a new VM and attaching that ISO fails: The ISO is not readable (says the vm console). The Disk Image from the test01 vm seems sounds (file has correct size, file-tool shows correct magic header), other files (configs) are readable. I did complete cluster reboot etc. gluster> volume heal vms info split-brain Brick node01:/gluster_bricks/vms/vms Status: Connected Number of entries in split-brain: 0 Brick node02:/gluster_bricks/vms/vms Status: Connected Number of entries in split-brain: 0 Brick node03:/gluster_bricks/vms/vms Status: Connected Number of entries in split-brain: 0 Brick node01:/gluster_bricks/vms/vms Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick node02:/gluster_bricks/vms/vms Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick node03:/gluster_bricks/vms/vms Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 How would I fix this issue? Anyone got a clue on how to proceed? -Chris. -- with kind regards, mit freundlichen Gruessen, Christian Reiss

1 1

Re: Question about HCI gluster_inventory.yml
by John Call 08 Jan '20

08 Jan '20

Simone, ovirt-users, I'm running into an issue with the ovirt.hosted_engine_setup role. I can't decide if my problem is 1) that the SSH connection to the HostedEngineLocal VM repeatedly fails due to that new VM not having an entry in ~/.ssh/known_hosts or 2) that I don't have some variable defined correctly. I can force my way past #1 by setting host_key_checking=false in ansible.cfg, but if I do that, I immediately get an error about an undefined hostvar variable... Can you point me in the right direction to debug this? I would really like to be able to rebuild my RHHI/RHV cluster using these gluster+ovirt roles. Here is the undefined variable error, and the variable files I'm including via the "ansible-playbook -e..." syntax: TASK [ovirt.hosted_engine_setup : Wait for the local VM] ***************************************************************************************** ok: [localhost -> rhvm.home.lab] TASK [ovirt.hosted_engine_setup : Add an entry for this host on /etc/hosts on the local VM] ****************************************************** fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. *The error was: u\"hostvars['rhhi1.home.lab']\" is undefined*\n\nThe error appears to be in '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/03_engine_initial_tasks.yml': line 8, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n timeout: 180\n - name: Add an entry for this host on /etc/hosts on the local VM\n ^ here\n"} Thank you, John Call Red Hat - Storage Domain Architect jcall(a)redhat.com (714) 267-8802 On Tue, Jan 7, 2020 at 12:27 AM Gobinda Das <godas(a)redhat.com> wrote: > Hi John, > Sorry misunderstood, I thought you are using cockpit based deployment. > Your gluster_inventory is correct for end to end deployment. > > On Tue, Jan 7, 2020 at 12:43 PM Gobinda Das <godas(a)redhat.com> wrote: > >> Hi John, >> I am seeing error: >> TASK [ovirt.hosted_engine_setup : Get local VM dir path] >> ***************************************************************************************** >> task path: >> /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/03_engine_initial_tasks.yml:57 >> fatal: [localhost]: FAILED! => { >> "msg": "The task includes an option with an undefined variable. The >> error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute >> 'he_local_vm_dir'\n\nThe error appears to be in >> '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/03_engine_initial_tasks.yml': >> line 57, column 7, but may\nbe elsewhere in the file depending on the exact >> syntax problem.\n\nThe offending line appears to be:\n\n delegate_to: >> \"{{ he_ansible_host_name }}\"\n - name: Get local VM dir path\n ^ >> here\n" >> } >> Adding @Simone Tiraboschi <stirabos(a)redhat.com> >> Simone do we need to explicitly specify he_local_vm_dir ? >> In gluster_inventory you can't mention other hosts and SD informations to >> auto add. For that you need to create separate file under: >> /usr/share/ovirt-hosted-engine-setup/gdeploy-inventory.yml >> and add entries there like: >> >> gluster: >> hosts: >> host2: >> host3: >> vars: >> storage_domains: >> [{"name":"data","host":"host1","address":"host1","path":"/data","mount_options":"backup-volfile-servers=host2:host3"},{"name":"vmstore","host":"host1","address":"host1","path":"/vmstore","mount_options":"backup-volfile-servers=host2:host3"}] >> >> Then hook will automatically read this file and perform action >> accordingly. >> >> On Tue, Jan 7, 2020 at 11:53 AM John Call <jcall(a)redhat.com> wrote: >> >>> Hi Gobinda, ovirt-users, >>> >>> I think I'm close, but I'm running into a timeout/error when the >>> ovirt.hosted-engine-setup tasks try to connect to the new >>> "HostedEngineLocal" VM. It seems to complain about host_key_checking. I >>> disabled that via ansible.cfg, but it fails on the next task (see short >>> logs.) Do you see anything obviously wrong in my inventory and json >>> files? I've attached the verbose playbook logs when I run this from the >>> first HCI host... >>> >>> [root@rhhi1 hc-ansible-deployment]# pwd >>> /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment >>> >>> [root@rhhi1 hc-ansible-deployment]# ansible-playbook -b -vvv -i >>> gluster_inventory.yml hc_deployment.yml >>> --extra-vars='@he_gluster_vars.json' | tee playbook.logs >>> >>> >>> Thank you, >>> John Call >>> Red Hat - Storage Domain Architect >>> jcall(a)redhat.com >>> (714) 267-8802 >>> >>> >>> On Wed, Dec 11, 2019 at 11:09 PM Gobinda Das <godas(a)redhat.com> wrote: >>> >>>> Hi John, >>>> You need to specify storage-fqdn(Which should mapped to storage >>>> network) and ovirtmgmt-fqdn (Which should mapped to frontend network) like >>>> this: >>>> hc_nodes: >>>> hosts: >>>> host1-STORAGE-fqdn: >>>> host2-STORAGE-fqdn: >>>> host3-STORAGE-fqdn: >>>> vars: >>>> cluster_nodes: >>>> - host1-STORAGE-fqdn >>>> - host2-STORAGE-fqdn >>>> - host3-STORAGE-fqdn >>>> gluster_features_hci_cluster: "{{ cluster_nodes }}" >>>> >>>> gluster: >>>> host2-ovirtmgmt-fqdn: >>>> host3-ovirtmgmt-fqdn: >>>> storage_domains: >>>> [{"name":"data","host":"host1-STORAGE-fqdn","address":"host1-STORAGE-fqdn","path":"/data","mount_options":"backup-volfile-servers=host2-STORAGE-fqdn:host3-STORAGE-fqdn"},{"name":"vmstore","host":"host1-STORAGE-fqdn","address":"host1-STORAGE-fqdn","path":"/vmstore","mount_options":"backup-volfile-servers=host2-STORAGE-fqdn:host3-STORAGE-fqdn"}] >>>> >>>> >>>> On Thu, Dec 12, 2019 at 2:47 AM John Call <jcall(a)redhat.com> wrote: >>>> >>>>> Hi ovirt-users, >>>>> >>>>> I'm trying to automate my HCI deployment, but can't figure out how to >>>>> specify multiple network interfaces in gluster_inventory.yml. My servers >>>>> have two NICs, one for ovirtmgmt (and everything else), and the other is >>>>> just for Gluster. How should I populate the inventory/vars file? Is this >>>>> correct? >>>>> >>>>> [root@rhhi1 hc-ansible-deployment]# pwd >>>>> /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment >>>>> >>>>> [root@rhhi1 hc-ansible-deployment]# cat gluster_inventory.yml >>>>> --lots of stuff omitted-- >>>>> hc_nodes: >>>>> hosts: >>>>> host1-STORAGE-fqdn: >>>>> host2-STORAGE-fqdn: >>>>> host3-STORAGE-fqdn: >>>>> vars: >>>>> cluster_nodes: >>>>> - host1-ovirtmgmt-fqdn >>>>> - host2-ovirtmgmt-fqdn >>>>> - host3-ovirtmgmt-fqdn >>>>> gluster_features_hci_cluster: "{{ cluster_nodes }}" >>>>> >>>>> gluster: >>>>> host2-STORAGE-fqdn: >>>>> host3-STORAGE-fqdn: >>>>> storage_domains: >>>>> [{"name":"data","host":"host1-STORAGE-fqdn","address":"host1-STORAGE-fqdn","path":"/data","mount_options":"backup-volfile-servers=host2-STORAGE-fqdn:host3-STORAGE-fqdn"},{"name":"vmstore","host":"host1-STORAGE-fqdn","address":"host1-STORAGE-fqdn","path":"/vmstore","mount_options":"backup-volfile-servers=host2-STORAGE-fqdn:host3-STORAGE-fqdn"}] >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/TEAHUX6LSPM5Y… >>>>> >>>> >>>> >>>> -- >>>> >>>> >>>> Thanks, >>>> Gobinda >>>> >>> >> >> -- >> >> >> Thanks, >> Gobinda >> > > > -- > > > Thanks, > Gobinda >

1 1