April 2019 - Users - oVirt List Archives

Re: Change IP Node and Manager
by Strahil 16 Apr '19

16 Apr '19

I think that you need to: 1. Put in maintenance 2. Unregister 3. Remove host 4. Change IP 5. Add host to oVirt again Please, someone more experienced to share any thoughts. Best Regards, Strahil NikolovOn Apr 15, 2019 22:49, "Sebastian Antunez N." <antunez.sebastian(a)gmail.com> wrote: > > Hello Guys > > Have 8 host with Ovirt 4.1 and need change the IP to all node. > > There is a procedure to make the change of IP, I have searched for information but I can not find a process to perform. > > Someone could help me to know how I can change the IP in the nodes, I know that I must put the nodes in maintenance, but I do not know if I should change the manager first, add an additional IP and then add the nodes, etc. > > Thanks for the help > > Sebastian

1 0

Re: hosted engine does not start
by Strahil 16 Apr '19

16 Apr '19

Try with the VNC console 'hosted-engine --add-console-password' Then connect on the IP:port that the command replies and check what is going on. Maybe, you will need a rescue DVD and mount all filesystems and dismount them. After that, just power it off and power it on regularly. If you can't use custom engine config, use the xml definition in the VDSM log. You will also need this alias: alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' , so you can use virsh freely (define/start/destroy). Best Regards, Strahil NikolovOn Apr 15, 2019 22:35, Stefan Wolf <shb256(a)gmail.com> wrote: > > Hello all, > > > > after a powerloss the hosted engine won’t start up anymore. > > I ‘ve the current ovirt installed. > > Storage is glusterfs und it is up and running > > > > It is trying to start up hosted engine but it does not work, but I can’t see where the problem is. > > > > [root@kvm320 ~]# hosted-engine --vm-status > > > > > > --== Host 1 status ==-- > > > > conf_on_shared_storage : True > > Status up-to-date : True > > Hostname : kvm380.durchhalten.intern > > Host ID : 1 > > Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Down"} > > Score : 1800 > > stopped : False > > Local maintenance : False > > crc32 : 3ad6d0bd > > local_conf_timestamp : 14594 > > Host timestamp : 14594 > > Extra metadata (valid at timestamp): > > metadata_parse_version=1 > > metadata_feature_version=1 > > timestamp=14594 (Mon Apr 15 21:25:12 2019) > > host-id=1 > > score=1800 > > vm_conf_refresh_time=14594 (Mon Apr 15 21:25:12 2019) > > conf_on_shared_storage=True > > maintenance=False > > state=GlobalMaintenance > > stopped=False > > > > > > --== Host 2 status ==-- > > > > conf_on_shared_storage : True > > Status up-to-date : True > > Hostname : kvm320.durchhalten.intern > > Host ID : 2 > > Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} > > Score : 0 > > stopped : False > > Local maintenance : False > > crc32 : e7d4840d > > local_conf_timestamp : 21500 > > Host timestamp : 21500 > > Extra metadata (valid at timestamp): > > metadata_parse_version=1 > > metadata_feature_version=1 > > timestamp=21500 (Mon Apr 15 21:25:22 2019) > > host-id=2 > > score=0 > > vm_conf_refresh_time=21500 (Mon Apr 15 21:25:22 2019) > > conf_on_shared_storage=True > > maintenance=False > > state=ReinitializeFSM > > stopped=False > > > > > > --== Host 3 status ==-- > > > > conf_on_shared_storage : True > > Status up-to-date : True > > Hostname : kvm360.durchhalten.intern > > Host ID : 3 > > Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} > > Score : 1800 > > stopped : False > > Local maintenance : False > > crc32 : cf9221cb > > local_conf_timestamp : 22121 > > Host timestamp : 22120 > > Extra metadata (valid at timestamp): > > metadata_parse_version=1 > > metadata_feature_version=1 > > timestamp=22120 (Mon Apr 15 21:25:18 2019) > > host-id=3 > > score=1800 > > vm_conf_refresh_time=22121 (Mon Apr 15 21:25:18 2019) > > conf_on_shared_storage=True > > maintenance=False > > state=GlobalMaintenance > > stopped=False > > > > [root@kvm320 ~]# virsh -r list > > Id Name Status > > ---------------------------------------------------- > > 6 HostedEngine laufend > > > > [root@kvm320 ~]# hosted-engine --console > > The engine VM is running on this host > > Verbunden mit der Domain: HostedEngine > > Escape-Zeichen ist ^] > > Fehler: Interner Fehler: Zeichengerät <null> kann nicht gefunden warden > > > > In engish it should be this > > > > [root@mgmt~]# hosted-engine --console > The engine VM is running on this host > Connected to domain HostedEngine > Escape character is ^] > error: internal error: cannot find character device > > > > This is in the log > > > > [root@kvm320 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log > > MainThread::INFO::2019-04-15 21:28:33,032::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 1800) > > MainThread::INFO::2019-04-15 21:28:43,050::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. > > MainThread::INFO::2019-04-15 21:28:43,165::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 1800) > > MainThread::INFO::2019-04-15 21:28:53,183::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. > > MainThread::INFO::2019-04-15 21:28:53,300::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 1800) > > MainThread::INFO::2019-04-15 21:29:03,317::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. > > MainThread::INFO::2019-04-15 21:29:03,434::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 1800) > > MainThread::INFO::2019-04-15 21:29:13,453::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. > > MainThread::INFO::2019-04-15 21:29:13,571::states::136::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Penalizing score by 1600 due to gateway status > > MainThread::INFO::2019-04-15 21:29:13,571::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 1800) > > MainThread::INFO::2019-04-15 21:29:22,589::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. > > MainThread::INFO::2019-04-15 21:29:22,712::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 1800) > > > > But it is not reachable over the network > > > > [root@kvm320 ~]# ping 192.168.200.211 > > PING 192.168.200.211 (192.168.200.211) 56(84) bytes of data. > > From 192.168.200.231 icmp_seq=1 Destination Host Unreachable > > From 192.168.200.231 icmp_seq=2 Destination Host Unreachable > > From 192.168.200.231 icmp_seq=3 Destination Host Unreachable > > From 192.168.200.231 icmp_seq=4 Destination Host Unreachable > > > > I tried to stop and start the vm again, but it didn’t helped > > > > Maybe someone can give me some advice how to get the hosted engine running again > > > > Thx by stefan

1 0

oVirt 4.3.2 - Cannot update Host via UI
by Strahil Nikolov 16 Apr '19

16 Apr '19

Hello guys, I have the following issue after I have successfully updated by engine from 4.3.1 to 4.3.2 - I canont update any host via the UI. The event log show startup of the update , but there is no process running on the Host, yum.log is not updated and engine log doesn't show anything meaningful. Any hint where to look for ? Thanks in advance. Best Regards,Strahil Nikolov

4 3

How to fix ovn apparent inconsistency?
by Gianluca Cecchi 16 Apr '19

16 Apr '19

Hello, passing from old manual to current OVN in 4.3.1 it seems I have some problems with OVN now. I cannot assign network on OVN to VM (powered on or off doesn't change). When I add//edit a vnic, they are not on the possible choices Environment composed by three hosts and one engine (external on vSphere). The mgmt network during time has been configured on network named ovirtmgmntZ2Z3 On engine it seems there are 2 switches for every defined ovn network (ovn192 and ovn172) Below some output of commands in case any inconsistency has remained and I can purge it. Thanks in advance. Gianluca - On manager ovmgr1: [root@ovmgr1 ~]# ovs-vsctl show eae54ff9-b86c-4050-8241-46f44336ba94 ovs_version: "2.10.1" [root@ovmgr1 ~]# [root@ovmgr1 ~]# ovn-nbctl show switch 32367d8a-460f-4447-b35a-abe9ea5187e0 (ovn192) port affc5570-3e5a-439c-9fdf-d75d6810e3a3 addresses: ["00:1a:4a:17:01:73"] port f639d541-2118-4c24-b478-b7a586eb170c addresses: ["00:1a:4a:17:01:75"] switch 6110649a-db2b-4de7-8fbc-601095cfe510 (ovn192) switch 64c4c17f-cd67-4e29-939e-2b952495159f (ovn172) port 32c348d9-12e9-4bcf-a43f-69338c887cfc addresses: ["00:1a:4a:17:01:72 dynamic"] port 3c77c2ea-de00-43f9-a5c5-9b3ffea5ec69 addresses: ["00:1a:4a:17:01:74 dynamic"] switch 04501f6b-3977-4ba1-9ead-7096768d796d (ovn172) port 0a2a47bc-ea0d-4f1d-8f49-ec903e519983 addresses: ["00:1a:4a:17:01:65 dynamic"] port 8fc7bed4-7663-4903-922b-05e490c6a5a1 addresses: ["00:1a:4a:17:01:64 dynamic"] port f2b64f89-b719-484c-ac02-2a1ac8eaacdb addresses: ["00:1a:4a:17:01:59 dynamic"] port f7389c88-1ea1-47c2-92fd-6beffb2e2190 addresses: ["00:1a:4a:17:01:58 dynamic"] [root@ovmgr1 ~]# - On host ov200 (10.4.192.32 on ovirtmgmntZ2Z3): [root@ov200 ~]# ovs-vsctl show ae0a1256-7250-46a2-a1b6-8f0ae6105c20 Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port "ovn-ddecf0-0" Interface "ovn-ddecf0-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.33"} Port "ovn-b8872a-0" Interface "ovn-b8872a-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.34"} ovs_version: "2.10.1" [root@ov200 ~]# - On host ov300 (10.4.192.33 on ovirtmgmntZ2Z3): [root@ov300 ~]# ovs-vsctl show f1a41e9c-16fb-4aa2-a386-2f366ade4d3c Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port "ovn-b8872a-0" Interface "ovn-b8872a-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.34"} Port "ovn-1dce5b-0" Interface "ovn-1dce5b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.32"} ovs_version: "2.10.1" [root@ov300 ~]# - On host ov301 (10.4.192.34 on ovirtmgmntZ2Z3): [root@ov301 ~]# ovs-vsctl show 3a38c5bb-0abf-493d-a2e6-345af8aedfe3 Bridge br-int fail_mode: secure Port "ovn-1dce5b-0" Interface "ovn-1dce5b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.32"} Port "ovn-ddecf0-0" Interface "ovn-ddecf0-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.33"} Port br-int Interface br-int type: internal ovs_version: "2.10.1" [root@ov301 ~]# In web admin gui: In network -> networks -> - ovn192 Id: 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5 VDSM Name: ovn192 External ID: 32367d8a-460f-4447-b35a-abe9ea5187e0 - ovn172 Id: 7546d5d3-a0e3-40d5-9d22-cf355da47d3a VDSM Name: ovn172 External ID: 64c4c17f-cd67-4e29-939e-2b952495159f

5 36

oVirt 4.3.2.1-1.el7 Errors at VM boot
by Wood Peter 15 Apr '19

15 Apr '19

Hi all, A few weeks ago I did a clean install of the latest oVirt-4.3.2 and imported some VMs from oVirt-3. Three nodes running oVirt Node and oVirt Engine installed on a separate system. I noticed that some times some VMs will boot successfully but the Web UI will still show "Powering UP" for days after the VM has been up. I can power down the VM and power back up and it may update the Web UI status to UP. While debugging the above issue I noticed that some VMs will trigger errors during boot. I can power on a VM on one node, see the errors below started happening every 4-5 seconds, then power down the VM, errors stop, then power up the VM on a different node without a problem. Another VM though may trigger the errors on the same node. Everything is very inconsistent. I can't find a pattern. I tried different VMs, different nodes, and I'm getting mixed results. Hopefully the errors will give some clue. Here is what I'm seeing scrolling every 4-5 seconds: ------------------------- On oVirt Node: ==> vdsm.log <== 2019-04-12 10:50:31,543-0700 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' ==> mom.log <== 2019-04-12 10:50:31,547 - mom.VdsmRpcBase - ERROR - Command Host.getAllVmStats with args {} failed: (code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}) ---------------------- On oVirt Engine 2019-04-12 10:50:35,692-07 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}] 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Failed in 'GetAllVmStatsVDS' method 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Command 'GetAllVmStatsVDSCommand(HostName = sdod-ovnode-03, VdsIdVDSCommandParametersBase:{hostId='12e38ad3-6327-4c94-8be4-88912d283729'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'appsList'"}, code = -32603 Thank you, -- Peter

2 2

Change IP Node and Manager
by Sebastian Antunez N. 15 Apr '19

15 Apr '19

Hello Guys Have 8 host with Ovirt 4.1 and need change the IP to all node. There is a procedure to make the change of IP, I have searched for information but I can not find a process to perform. Someone could help me to know how I can change the IP in the nodes, I know that I must put the nodes in maintenance, but I do not know if I should change the manager first, add an additional IP and then add the nodes, etc. Thanks for the help Sebastian

1 0

Poor I/O Performance (again...)
by Jim Kusznir 15 Apr '19

15 Apr '19

Hi all: I've had I/O performance problems pretty much since the beginning of using oVirt. I've applied several upgrades as time went on, but strangely, none of them have alleviated the problem. VM disk I/O is still very slow to the point that running VMs is often painful; it notably affects nearly all my VMs, and makes me leary of starting any more. I'm currently running 12 VMs and the hosted engine on the stack. My configuration started out with 1Gbps networking and hyperconverged gluster running on a single SSD on each node. It worked, but I/O was painfully slow. I also started running out of space, so I added an SSHD on each node, created another gluster volume, and moved VMs over to it. I also ran that on a dedicated 1Gbps network. I had recurring disk failures (seems that disks only lasted about 3-6 months; I warrantied all three at least once, and some twice before giving up). I suspect the Dell PERC 6/i was partly to blame; the raid card refused to see/acknowledge the disk, but plugging it into a normal PC showed no signs of problems. In any case, performance on that storage was notably bad, even though the gig-e interface was rarely taxed. I put in 10Gbps ethernet and moved all the storage on that none the less, as several people here said that 1Gbps just wasn't fast enough. Some aspects improved a bit, but disk I/O is still slow. And I was still having problems with the SSHD data gluster volume eating disks, so I bought a dedicated NAS server (supermicro 12 disk dedicated FreeNAS NFS storage system on 10Gbps ethernet). Set that up. I found that it was actually FASTER than the SSD-based gluster volume, but still slow. Lately its been getting slower, too...Don't know why. The FreeNAS server reports network loads around 4MB/s on its 10Gbe interface, so its not network constrained. At 4MB/s, I'd sure hope the 12 spindle SAS interface wasn't constrained either..... (and disk I/O operations on the NAS itself complete much faster). So, running a test on my NAS against an ISO file I haven't accessed in months: # dd if=en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso of=/dev/null bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 2.459501 secs (213168465 bytes/sec) Running it on one of my hosts: root@unifi:/home/kusznir# time dd if=/dev/sda of=/dev/null bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 7.21337 s, 72.7 MB/s (I don't know if this is a true apples to apples comparison, as I don't have a large file inside this VM's image). Even this is faster than I often see. I have a VoIP Phone server running as a VM. Voicemail and other recordings usually fail due to IO issues opening and writing the files. Often, the first 4 or so seconds of the recording is missed; sometimes the entire thing just fails. I didn't use to have this problem, but its definately been getting worse. I finally bit the bullet and ordered a physical server dedicated for my VoIP System...But I still want to figure out why I'm having all these IO problems. I read on the list of people running 30+ VMs...I feel that my IO can't take any more VMs with any semblance of reliability. We have a Quickbooks server on here too (windows), and the performance is abysmal; my CPA is charging me extra because of all the lost staff time waiting on the system to respond and generate reports..... I'm at my whits end...I started with gluster on SSD with 1Gbps network, migrated to 10Gbps network, and now to dedicated high performance NAS box over NFS, and still have performance issues.....I don't know how to troubleshoot the issue any further, but I've never had these kinds of issues when I was playing with other VM technologies. I'd like to get to the point where I can resell virtual servers to customers, but I can't do so with my current performance levels. I'd greatly appreciate help troubleshooting this further. --Jim

3 6

Re: Tuning Gluster Writes
by Strahil 15 Apr '19

15 Apr '19

Hi, What is your dirty cache settings on the gluster servers ? Best Regards, Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter <alex(a)triadic.us> wrote: > > I have 8 machines acting as gluster servers. They each have 12 drives > raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as > one). > > They connect to the compute hosts and to each other over lacp'd 10GB > connections split across two cisco nexus switched with VPC. > > Gluster has the following set. > > performance.write-behind-window-size: 4MB > performance.flush-behind: on > performance.stat-prefetch: on > server.event-threads: 4 > client.event-threads: 8 > performance.io-thread-count: 32 > network.ping-timeout: 30 > cluster.granular-entry-heal: enable > performance.strict-o-direct: on > storage.owner-gid: 36 > storage.owner-uid: 36 > features.shard: on > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > auth.allow: * > user.cifs: off > transport.address-family: inet > nfs.disable: off > performance.client-io-threads: on > > > I have the following sysctl values on gluster client and servers, using > libgfapi, MTU 9K > > net.core.rmem_max = 134217728 > net.core.wmem_max = 134217728 > net.ipv4.tcp_rmem = 4096 87380 134217728 > net.ipv4.tcp_wmem = 4096 65536 134217728 > net.core.netdev_max_backlog = 300000 > net.ipv4.tcp_moderate_rcvbuf =1 > net.ipv4.tcp_no_metrics_save = 1 > net.ipv4.tcp_congestion_control=htcp > > reads with this setup are perfect, benchmarked in VM to be about 770MB/s > sequential with disk access times of < 1ms. Writes on the other hand are > all over the place. They peak around 320MB/s sequential write, which is > what i expect but it seems as if there is some blocking going on. > > During the write test i will hit 320MB/s briefly, then 0MB/s as disk > access time shoot to over 3000ms, then back to 320MB/s. It averages out > to about 110MB/s afterwards. > > Gluster version is 3.12.15 ovirt is 4.2.7.5 > > Any ideas on what i could tune to eliminate or minimize that blocking? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGIC…

4 9

oVirt 4.3.2 missing/wrong status of VM
by Strahil Nikolov 15 Apr '19

15 Apr '19

As I couldn't find the exact mail thread, I'm attaching my /usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py which fixes the missing/wrong status of VMs. You will need to restart vdsmd (I'm not sure how safe is that with running guests) in order to start working. Best Regards,Strahil Nikolov

2 1

Second host fail to activate (hosted-engine)
by Ricardo Alonso 15 Apr '19

15 Apr '19

After installing the second host via the web gui (4.3.2.1-1.el7), it fails to activate telling that wasn't possible to connect to the storage pool default (glusterfs). Those are the logs: vdsm.log 2019-04-09 15:54:07,409-0400 INFO (Reactor thread) [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:58130 (protocoldetector:61) 2019-04-09 15:54:07,419-0400 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::1:58130 (protocoldetector:125) 2019-04-09 15:54:07,419-0400 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2019-04-09 15:54:07,420-0400 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2019-04-09 15:54:07,461-0400 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2019-04-09 15:54:07,466-0400 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2019-04-09 15:54:07,469-0400 INFO (jsonrpc/0) [vdsm.api] START getStorageDomainInfo(sdUUID=u'd99fb087-66d5-4adf-9c0c-80e60de17917', options=None) from=::1,58130, task_id=00c843c2-ab43-4813-9ded-29f6742c33b2 (api:48) 2019-04-09 15:54:07,484-0400 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo error='VERSION' from=::1,58130, task_id=00c843c2-ab43-4813-9ded-29f6742c33b2 (api:52) 2019-04-09 15:54:07,484-0400 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='00c843c2-ab43-4813-9ded-29f6742c33b2') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getStorageDomainInfo File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2741, in getStorageDomainInfo dom = self.validateSdUUID(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 305, in validateSdUUID sdDom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterSD.py", line 56, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 394, in __init__ manifest = self.manifestClass(domainPath) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 179, in __init__ sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 332, in __init__ self._domainLock = self._makeDomainLock() File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 553, in _makeDomainLock domVersion = self.getVersion() File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 424, in getVersion return self.getMetaParam(DMDK_VERSION) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 421, in getMetaParam return self._metadata[key] File "/usr/lib/python2.7/site-packages/vdsm/storage/persistent.py", line 91, in __getitem__ return dec(self._dict[key]) File "/usr/lib/python2.7/site-packages/vdsm/storage/persistent.py", line 202, in __getitem__ return self._metadata[key] KeyError: 'VERSION' 2019-04-09 15:54:07,484-0400 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='00c843c2-ab43-4813-9ded-29f6742c33b2') aborting: Task is aborted: u"'VERSION'" - code 100 (task:1181) 2019-04-09 15:54:07,484-0400 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getStorageDomainInfo error='VERSION' (dispatcher:87) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error KeyError: 'VERSION' 2019-04-09 15:54:07,484-0400 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getInfo failed (error 350) in 0.01 seconds (__init__:312) 2019-04-09 15:54:07,502-0400 INFO (jsonrpc/3) [vdsm.api] START connectStorageServer(domType=7, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d', u'vfs_type': u'glusterfs', u'connection': u'poseidon:/engine', u'user': u'kvm'}], options=None) from=::1,58130, task_id=d71268fe-0088-44e1-99e8-7bcc868b3b2e (api:48) 2019-04-09 15:54:07,521-0400 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}]} from=::1,58130, task_id=d71268fe-0088-44e1-99e8-7bcc868b3b2e (api:54) 2019-04-09 15:54:07,521-0400 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.02 seconds (__init__:312) 2019-04-09 15:54:07,533-0400 INFO (jsonrpc/4) [vdsm.api] START getStorageDomainStats(sdUUID=u'd99fb087-66d5-4adf-9c0c-80e60de17917', options=None) from=::1,58130, task_id=47c23532-80c9-4186-948b-be9ed264bfbf (api:48) agent.log MainThread::INFO::2019-04-09 16:07:23,647::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.1 started MainThread::INFO::2019-04-09 16:07:23,699::hosted_engine::244::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: potential.o2pos.com.br MainThread::INFO::2019-04-09 16:07:23,825::hosted_engine::524::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2019-04-09 16:07:23,827::brokerlink::77::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.8.1'} MainThread::ERROR::2019-04-09 16:07:23,828::hosted_engine::540::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2019-04-09 16:07:23,828::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 413, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 537, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'ping', options: {'addr': '192.168.8.1'}] MainThread::ERROR::2019-04-09 16:07:23,829::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2019-04-09 16:07:23,829::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down brocker.log MainThread::INFO::2019-04-09 16:08:00,892::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.1 started MainThread::INFO::2019-04-09 16:08:00,892::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors MainThread::INFO::2019-04-09 16:08:00,893::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2019-04-09 16:08:00,895::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2019-04-09 16:08:00,895::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2019-04-09 16:08:00,896::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2019-04-09 16:08:00,896::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-load MainThread::INFO::2019-04-09 16:08:00,896::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2019-04-09 16:08:00,897::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor ping MainThread::INFO::2019-04-09 16:08:00,897::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain MainThread::INFO::2019-04-09 16:08:00,897::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2019-04-09 16:08:00,898::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2019-04-09 16:08:00,898::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2019-04-09 16:08:00,899::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2019-04-09 16:08:00,899::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-load MainThread::INFO::2019-04-09 16:08:00,899::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2019-04-09 16:08:00,900::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor ping MainThread::INFO::2019-04-09 16:08:00,900::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain MainThread::INFO::2019-04-09 16:08:00,900::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::INFO::2019-04-09 16:08:00,957::storage_backends::345::ovirt_hosted_engine_ha.lib.storage_backends::(connect) Connecting the storage MainThread::INFO::2019-04-09 16:08:00,958::storage_server::349::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2019-04-09 16:08:00,993::storage_server::356::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2019-04-09 16:08:01,025::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::WARNING::2019-04-09 16:08:01,322::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: Command Image.prepare with args {'imageID': 'e525f96e-ffa3-43a8-a368-d473f064944a', 'storagepoolID': '00000000-0000-0000-0000-000000000000', 'volumeID': '12c2075c-4796-4185-b7f3-ed9f366d95ef', 'storagedomainID': 'd99fb087-66d5-4adf-9c0c-80e60de17917'} failed: (code=100, message='VERSION')

2 2

Live storage migration is failing in 4.2.8
by Ladislav Humenik 15 Apr '19

15 Apr '19

Hello, we have recently updated few ovirts from 4.2.5 to 4.2.8 version (actually 9 ovirt engine nodes), where the live storage migration stopped to work, and leave auto-generated snapshot behind. If we power the guest VM down, the migration works as expected. Is there a known bug for this? Shall we open a new one? Setup: ovirt - Dell PowerEdge R630 - CentOS Linux release 7.6.1810 (Core) - ovirt-engine-4.2.8.2-1.el7.noarch - kernel-3.10.0-957.10.1.el7.x86_64 hypervisors - Dell PowerEdge R640 - CentOS Linux release 7.6.1810 (Core) - kernel-3.10.0-957.10.1.el7.x86_64 - vdsm-4.20.46-1.el7.x86_64 - libvirt-5.0.0-1.el7.x86_64 - qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64 storage domain - netapp NFS share logs are attached -- Ladislav Humenik System administrator

3 3

Global maintenance and fencing of hosts
by Andreas Elvers 15 Apr '19

15 Apr '19

I am wondering whether global maintenance inhibits fencing of non-responsive hosts. Is this so? Background: I plan on migrating the engine from one cluster to another. I understand this means to backup/restore the engine. While migrating the engine it is shut down and all VMs will continue running. This is good. When starting the engine in the new location, I really don't want the engine to fence any host on its own, because of reasons I can not yet know. So is global maintenance enough to suppress fencing, or do I have to deactivate fencing on all hosts?

3 2

VDSM command CreateStoragePoolVDS faile
by fangjian＠linuxtrend.cn 14 Apr '19

14 Apr '19

Hi, I tried to setup Ovirt in my environment but encountered some problems in data domain creation. oVirt Manager Version 4.3.2.1-1.el7 oVirt Node Version 4.3.2 1. Create new data center & cluster; ---- successful 2. Create new host 'centos-node-01' in cluster; ---- successful 3. Prepare the NFS storage; ---- successful 4. Tried to create new datadomain and attach NFS storage to host 'centos-node-01' but failed. Message: VDSM centos-node-01 command CreateStoragePoolVDS failed: Cannot acquire host id: (u'ec225640-f05e-4a9d-bdc4-5219065704ec', SanlockException(2, 'Sanlock lockspace add failure', 'No such file or directory')) How can I fix the problem and attach the NFS storage to new datadomain?

1 0

Re: Expand existing gluster storage in ovirt 4.2/4.3
by Strahil 14 Apr '19

14 Apr '19

Just add new servers to the gluster cluster (by 3). Then just install them from the oVirt HostedEngine. Best Regards, Strahil NikolovOn Apr 11, 2019 15:49, adrianquintero(a)gmail.com wrote: > > Would you know of any documentation that I could follow for that type of setup? > > I have read quite a bit about oVIrt and Hyperconverged but I have only found 3 node setup examples, nobody seems to go past that or at least I've yet to find one. I already have a 3 node setup with hyperconverged and working as it should with no issues and tested fail scenarios, but I need to have an environment with at least 12 servers and be able to maintain the correct fail scenarios with storage (gluster), however I cant seem to figure out the proper steps to achieve this. > > So if anyone can point me in the right direction it will help a lot, and once I test I should be able to provide back any knowledge that I gain with such type of setup. > > > thanks again. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PJIRZGP6NQ4RK…

2 1

Host reinstall: ModuleNotFoundError: No module named 'rpmUtils'
by John Florian 14 Apr '19

14 Apr '19

After mucking around trying to use jumbo MTU for my iSCSI storage nets (which apparently I can't do because my Cisco 3560 switch only supports 1500 max for its vlan interfaces) I got one of my Hosts screwed up. I likely could rebuild it from scratch but I suspect that's overkill. I simply tried to do a reinstall via the GUI. That fails. Looking at the ovirt-host-deploy log I see several tracebacks with $SUBJECT. Since Python pays my bills I figure this is an easy fix. Except ... I see this on the host: $ rpm -qf /usr/lib/python2.7/site-packages/rpmUtils/ yum-3.4.3-161.el7.centos.noarch $ python Python 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Tab completion has been enabled. >>> import rpmUtils >>> I'm guessing this must mean the tracebacks are from Python 3 since I can clearly see the module doesn't exist for either Python 3.4 or 3.6. So this smells like a packaging bug somehow related to upgrading from 4.2. I mean, I can't imagine a brand new install fails this blatantly. Either that or this import error has nothing to do with my reinstall failure. -- John Florian

2 3

Bug 1666795 - Related? - VM's don't start after shutdown on FCP
by nardusg＠gmail.com 14 Apr '19

14 Apr '19

Hi There Wonder if this issue is related to our problem and if there is a way around it. We upgraded from 4.2.8. to 4.3.2. Now when we start some of the VM's fail to start. You need to deattach the disks, create new VM, reattach the disks to the new VM and then the new VM starts. Thanks Nar

4 8

Unable to start vdsm, upgrade 4.0 to 4.1
by Todd Barton 13 Apr '19

13 Apr '19

Looking for some help/suggestions to correct an issue I'm having. I have a 3 host HA setup running a hosted-engine and gluster storage. The hosts are identical hardware configurations and have been running for several years very solidly. I was performing an upgrade to 4.1. 1st host when fine. The second upgrade didn't go well...On server reboot, it went into kernel panic and I had to load previous kernel to diagnose. I couldn't get it out of panic and I had to revert the system to the previous kernel which was a big PITA. I updated it to current and verified installation of ovirt/vdsm. Everything seemed to be ok, but vdsm won't start. Gluster is working fine. It appears I have a authentication issue with libvirt. I'm getting the message "libvirt: XML-RPC error : authentication failed: authentication failed" which seems to be the core issue. I've looked at all the past issues/resolutions to this issue and tried them, but I can't get it to work. For example, I do a vdsm-tool configure --force and I get this... Checking configuration status... abrt is already configured for vdsm lvm is configured for vdsm libvirt is already configured for vdsm SUCCESS: ssl configured to true. No conflicts Current revision of multipath.conf detected, preserving Running configure... Reconfiguration of abrt is done. Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 219, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python2.7/site-packages/vdsm/tool/__init__.py", line 38, in wrapper func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py", line 141, in configure _configure(c) File "/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py", line 88, in _configure getattr(module, 'configure', lambda: None)() File "/usr/lib/python2.7/site-packages/vdsm/tool/configurators/passwd.py", line 68, in configure configure_passwd() File "/usr/lib/python2.7/site-packages/vdsm/tool/configurators/passwd.py", line 98, in configure_passwd raise RuntimeError("Set password failed: %s" % (err,)) RuntimeError: Set password failed: ['saslpasswd2: invalid parameter supplied'] ...and help would be greatly appreciated. I'm not a linux/ovirt expert by any means, but I desperately need to get this setup back to being stable. This happened many months ago and I gave up fixing, but I really need to get this back online again. Thank you Todd Barton

2 1

Re: Unable to start vdsm, upgrade 4.0 to 4.1
by Strahil 13 Apr '19

13 Apr '19

The fastest (windows style) approach is to completely wipe the host and do a reinstall -> install in vdsm and so on. You should consider oVirt Node. Another one that comes to my mind is to do: yum history yum history rollback <id> reboot repeat the upgrade. For now, I'm planing to do gluster snapshots (all machines on volume -> stopped) before major upgrades, as this is the fastest recovery approach. Actually oVirt Node uses thin lvm snapshots to guarantee fast rollback. Best Regards, Strahil NikolovOn Apr 13, 2019 06:44, Todd Barton <tcbarton(a)ipvoicedatasystems.com> wrote: > > Looking for some help/suggestions to correct an issue I'm having. I have a 3 host HA setup running a hosted-engine and gluster storage. The hosts are identical hardware configurations and have been running for several years very solidly. I was performing an upgrade to 4.1. 1st host when fine. The second upgrade didn't go well...On server reboot, it went into kernel panic and I had to load previous kernel to diagnose. > > I couldn't get it out of panic and I had to revert the system to the previous kernel which was a big PITA. I updated it to current and verified installation of ovirt/vdsm. Everything seemed to be ok, but vdsm won't start. Gluster is working fine. It appears I have a authentication issue with libvirt. I'm getting the message "libvirt: XML-RPC error : authentication failed: authentication failed" which seems to be the core issue. > > I've looked at all the past issues/resolutions to this issue and tried them, but I can't get it to work. For example, I do a vdsm-tool configure --force and I get this... > > Checking configuration status... > > abrt is already configured for vdsm > lvm is configured for vdsm > libvirt is already configured for vdsm > SUCCESS: ssl configured to true. No conflicts > Current revision of multipath.conf detected, preserving > > Running configure... > Reconfiguration of abrt is done. > Traceback (most recent call last): > File "/usr/bin/vdsm-tool", line 219, in main > return tool_command[cmd]["command"](*args) > File "/usr/lib/python2.7/site-packages/vdsm/tool/__init__.py", line 38, in wrapper > func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py", line 141, in configure > _configure(c) > File "/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py", line 88, in _configure > getattr(module, 'configure', lambda: None)() > File "/usr/lib/python2.7/site-packages/vdsm/tool/configurators/passwd.py", line 68, in configure > configure_passwd() > File "/usr/lib/python2.7/site-packages/vdsm/tool/configurators/passwd.py", line 98, in configure_passwd > raise RuntimeError("Set password failed: %s" % (err,)) > RuntimeError: Set password failed: ['saslpasswd2: invalid parameter supplied'] > > ...and help would be greatly appreciated. I'm not a linux/ovirt expert by any means, but I desperately need to get this setup back to being stable. This happened many months ago and I gave up fixing, but I really need to get this back online again. > > Thank you > > Todd Barton > > > > >

2 1

Tuning Gluster Writes
by Alex McWhirter 12 Apr '19

12 Apr '19

I have 8 machines acting as gluster servers. They each have 12 drives raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as one). They connect to the compute hosts and to each other over lacp'd 10GB connections split across two cisco nexus switched with VPC. Gluster has the following set. performance.write-behind-window-size: 4MB performance.flush-behind: on performance.stat-prefetch: on server.event-threads: 4 client.event-threads: 8 performance.io-thread-count: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: off performance.client-io-threads: on I have the following sysctl values on gluster client and servers, using libgfapi, MTU 9K net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 300000 net.ipv4.tcp_moderate_rcvbuf =1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_congestion_control=htcp reads with this setup are perfect, benchmarked in VM to be about 770MB/s sequential with disk access times of < 1ms. Writes on the other hand are all over the place. They peak around 320MB/s sequential write, which is what i expect but it seems as if there is some blocking going on. During the write test i will hit 320MB/s briefly, then 0MB/s as disk access time shoot to over 3000ms, then back to 320MB/s. It averages out to about 110MB/s afterwards. Gluster version is 3.12.15 ovirt is 4.2.7.5 Any ideas on what i could tune to eliminate or minimize that blocking?

1 0

Migrate self-hosted engine between cluster
by raul.caballero.girol＠gmail.com 12 Apr '19

12 Apr '19

Hello everibody, Iḿ a noob with ovirt and I have a problem. I have deployed a new enviroment with a self-hosted engine. Now, when I am learning about ovirt, I think that my engine should be in another cluster. How I can move my manage from one cluster to other?. Cheers

3 3

Re: Cannot allocate and run VM from VM-Pool. There are no available VMs in the VM-Pool
by nicolas＠devels.es 12 Apr '19

12 Apr '19

Are the VMs from the pool 'up'? If so, no assignation can be done unless they are powered off. El 2019-04-12 14:31, Florian Rädler escribió: > I am getting the following Error after a Pool was generated and > migrated to another host. > > START_POOL fehlgeschlagen [Cannot allocate and run VM from VM-Pool. > There are no available VMs in the VM-Pool.] > > No user is connected to any of the running VMs. What can I do to solve > this problem? > > ------------------------- > > Pflichtangaben anzeigen [1] > > Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie > hier: http://www.deutschebahn.com/de/konzern/datenschutz [2] > > Links: > ------ > [1] http://www.deutschebahn.com/pflichtangaben/20190408 > [2] http://www.deutschebahn.com/de/konzern/datenschutz > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UL2KJKETQYFZR…

1 0

Cannot allocate and run VM from VM-Pool. There are no available VMs in the VM-Pool
by Florian Rädler 12 Apr '19

12 Apr '19

I am getting the following Error after a Pool was generated and migrated to another host. START_POOL fehlgeschlagen [Cannot allocate and run VM from VM-Pool. There are no available VMs in the VM-Pool.] No user is connected to any of the running VMs. What can I do to solve this problem? ________________________________ Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20190408> Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz

1 0

[ANN] oVirt 4.3.3 Fourth Release Candidate is now available
by Sandro Bonazzola 12 Apr '19

12 Apr '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.3 Fourth Release Candidate, as of April 12th, 2019. This update is a release candidate of the third in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance will be available soon - oVirt Node will be available soon[2] Additional Resources: * Read more about the oVirt 4.3.3 release highlights: http://www.ovirt.org/release/4.3.3/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.3/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

Re: [Gluster-users] Gluster snapshot fails
by Strahil Nikolov 12 Apr '19

12 Apr '19

Hello All, I have tried to enable debug and see the reason for the issue. Here is the relevant glusterd.log: [2019-04-12 07:56:54.526508] E [MSGID: 106077] [glusterd-snapshot.c:1882:glusterd_is_thinp_brick] 0-management: Failed to get pool name for device systemd-1 [2019-04-12 07:56:54.527509] E [MSGID: 106121] [glusterd-snapshot.c:2523:glusterd_snapshot_create_prevalidate] 0-management: Failed to pre validate [2019-04-12 07:56:54.527525] E [MSGID: 106024] [glusterd-snapshot.c:2547:glusterd_snapshot_create_prevalidate] 0-management: Snapshot is supported only for thin provisioned LV. Ensure that all bricks of isos are thinly provisioned LV. [2019-04-12 07:56:54.527539] W [MSGID: 106029] [glusterd-snapshot.c:8613:glusterd_snapshot_prevalidate] 0-management: Snapshot create pre-validation failed [2019-04-12 07:56:54.527552] W [MSGID: 106121] [glusterd-mgmt.c:147:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed [2019-04-12 07:56:54.527568] E [MSGID: 106121] [glusterd-mgmt.c:1015:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node [2019-04-12 07:56:54.527583] E [MSGID: 106121] [glusterd-mgmt.c:2377:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed here is the output of lvscan & lvs: [root@ovirt1 ~]# lvscan ACTIVE '/dev/gluster_vg_md0/my_vdo_thinpool' [9.86 TiB] inherit ACTIVE '/dev/gluster_vg_md0/gluster_lv_data' [500.00 GiB] inherit ACTIVE '/dev/gluster_vg_md0/gluster_lv_isos' [50.00 GiB] inherit ACTIVE '/dev/gluster_vg_ssd/my_ssd_thinpool' [168.59 GiB] inherit ACTIVE '/dev/gluster_vg_ssd/gluster_lv_engine' [40.00 GiB] inherit ACTIVE '/dev/centos_ovirt1/swap' [6.70 GiB] inherit ACTIVE '/dev/centos_ovirt1/home' [1.00 GiB] inherit ACTIVE '/dev/centos_ovirt1/root' [60.00 GiB] inherit [root@ovirt1 ~]# lvs --noheadings -o pool_lv my_vdo_thinpool my_vdo_thinpool my_ssd_thinpool [root@ovirt1 ~]# ssh ovirt2 "lvscan;lvs --noheadings -o pool_lv" ACTIVE '/dev/gluster_vg_md0/my_vdo_thinpool' [<9.77 TiB] inherit ACTIVE '/dev/gluster_vg_md0/gluster_lv_data' [500.00 GiB] inherit ACTIVE '/dev/gluster_vg_md0/gluster_lv_isos' [50.00 GiB] inherit ACTIVE '/dev/gluster_vg_ssd/my_ssd_thinpool' [<161.40 GiB] inherit ACTIVE '/dev/gluster_vg_ssd/gluster_lv_engine' [40.00 GiB] inherit ACTIVE '/dev/centos_ovirt2/root' [15.00 GiB] inherit ACTIVE '/dev/centos_ovirt2/home' [1.00 GiB] inherit ACTIVE '/dev/centos_ovirt2/swap' [16.00 GiB] inherit my_vdo_thinpool my_vdo_thinpool my_ssd_thinpool [root@ovirt1 ~]# ssh ovirt3 "lvscan;lvs --noheadings -o pool_lv" ACTIVE '/dev/gluster_vg_sda3/gluster_thinpool_sda3' [41.00 GiB] inherit ACTIVE '/dev/gluster_vg_sda3/gluster_lv_data' [15.00 GiB] inherit ACTIVE '/dev/gluster_vg_sda3/gluster_lv_isos' [15.00 GiB] inherit ACTIVE '/dev/gluster_vg_sda3/gluster_lv_engine' [15.00 GiB] inherit ACTIVE '/dev/centos_ovirt3/root' [20.00 GiB] inherit ACTIVE '/dev/centos_ovirt3/home' [1.00 GiB] inherit ACTIVE '/dev/centos_ovirt3/swap' [8.00 GiB] inherit gluster_thinpool_sda3 gluster_thinpool_sda3 gluster_thinpool_sda3 I am mounting my bricks via systemd , as I have issues with bricks being started before VDO. [root@ovirt1 ~]# findmnt /gluster_bricks/isos TARGET SOURCE FSTYPE OPTIONS /gluster_bricks/isos systemd-1 autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21843 /gluster_bricks/isos /dev/mapper/gluster_vg_md0-gluster_lv_isos xfs rw,noatime,nodiratime,seclabel,attr2,inode64,noquota [root@ovirt1 ~]# ssh ovirt2 "findmnt /gluster_bricks/isos " TARGET SOURCE FSTYPE OPTIONS /gluster_bricks/isos systemd-1 autofs rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14279 /gluster_bricks/isos /dev/mapper/gluster_vg_md0-gluster_lv_isos xfs rw,noatime,nodiratime,seclabel,attr2,inode64,noquota [root@ovirt1 ~]# ssh ovirt3 "findmnt /gluster_bricks/isos " TARGET SOURCE FSTYPE OPTIONS /gluster_bricks/isos systemd-1 autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17770 /gluster_bricks/isos /dev/mapper/gluster_vg_sda3-gluster_lv_isos xfs rw,noatime,nodiratime,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=1024,noquota [root@ovirt1 ~]# grep "gluster_bricks" /proc/mounts systemd-1 /gluster_bricks/data autofs rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21513 0 0 systemd-1 /gluster_bricks/engine autofs rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21735 0 0 systemd-1 /gluster_bricks/isos autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21843 0 0 /dev/mapper/gluster_vg_ssd-gluster_lv_engine /gluster_bricks/engine xfs rw,seclabel,noatime,nodiratime,attr2,inode64,sunit=256,swidth=256,noquota 0 0 /dev/mapper/gluster_vg_md0-gluster_lv_isos /gluster_bricks/isos xfs rw,seclabel,noatime,nodiratime,attr2,inode64,noquota 0 0 /dev/mapper/gluster_vg_md0-gluster_lv_data /gluster_bricks/data xfs rw,seclabel,noatime,nodiratime,attr2,inode64,noquota 0 0 Obviously , gluster is catching "systemd-1" as a device and tries to check if it's a thin LV.Where should I open a bug for that ? P.S.: Adding oVirt User list. Best Regards,Strahil Nikolov В четвъртък, 11 април 2019 г., 4:00:31 ч. Гринуич-4, Strahil Nikolov <hunter86_bg(a)yahoo.com> написа: Hi Rafi, thanks for your update. I have tested again with another gluster volume.[root@ovirt1 glusterfs]# gluster volume info isos Volume Name: isos Type: Replicate Volume ID: 9b92b5bd-79f5-427b-bd8d-af28b038ed2a Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: ovirt1:/gluster_bricks/isos/isos Brick2: ovirt2:/gluster_bricks/isos/isos Brick3: ovirt3.localdomain:/gluster_bricks/isos/isos (arbiter) Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.enable-shared-storage: enable Command run: logrotate -f glusterfs ; logrotate -f glusterfs-georep; gluster snapshot create isos-snap-2019-04-11 isos description TEST Logs:[root@ovirt1 glusterfs]# cat cli.log [2019-04-11 07:51:02.367453] I [cli.c:769:main] 0-cli: Started running gluster with version 5.5 [2019-04-11 07:51:02.486863] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2019-04-11 07:51:02.556813] E [cli-rpc-ops.c:11293:gf_cli_snapshot] 0-cli: cli_to_glusterd for snapshot failed [2019-04-11 07:51:02.556880] I [input.c:31:cli_batch] 0-: Exiting with: -1 [root@ovirt1 glusterfs]# cat glusterd.log [2019-04-11 07:51:02.553357] E [MSGID: 106024] [glusterd-snapshot.c:2547:glusterd_snapshot_create_prevalidate] 0-management: Snapshot is supported only for thin provisioned LV. Ensure that all bricks of isos are thinly provisioned LV. [2019-04-11 07:51:02.553365] W [MSGID: 106029] [glusterd-snapshot.c:8613:glusterd_snapshot_prevalidate] 0-management: Snapshot create pre-validation failed [2019-04-11 07:51:02.553703] W [MSGID: 106121] [glusterd-mgmt.c:147:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed [2019-04-11 07:51:02.553719] E [MSGID: 106121] [glusterd-mgmt.c:1015:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node My LVs hosting the bricks are:[root@ovirt1 ~]# lvs gluster_vg_md0 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert gluster_lv_data gluster_vg_md0 Vwi-aot--- 500.00g my_vdo_thinpool 35.97 gluster_lv_isos gluster_vg_md0 Vwi-aot--- 50.00g my_vdo_thinpool 52.11 my_vdo_thinpool gluster_vg_md0 twi-aot--- 9.86t 2.04 11.45 [root@ovirt1 ~]# ssh ovirt2 "lvs gluster_vg_md0" LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert gluster_lv_data gluster_vg_md0 Vwi-aot--- 500.00g my_vdo_thinpool 35.98 gluster_lv_isos gluster_vg_md0 Vwi-aot--- 50.00g my_vdo_thinpool 25.94 my_vdo_thinpool gluster_vg_md0 twi-aot--- <9.77t 1.93 11.39 [root@ovirt1 ~]# ssh ovirt3 "lvs gluster_vg_sda3" LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert gluster_lv_data gluster_vg_sda3 Vwi-aotz-- 15.00g gluster_thinpool_sda3 0.17 gluster_lv_engine gluster_vg_sda3 Vwi-aotz-- 15.00g gluster_thinpool_sda3 0.16 gluster_lv_isos gluster_vg_sda3 Vwi-aotz-- 15.00g gluster_thinpool_sda3 0.12 gluster_thinpool_sda3 gluster_vg_sda3 twi-aotz-- 41.00g 0.16 1.58 As you can see - all bricks are thin LV and space is not the issue. Can someone hint me how to enable debug , so gluster logs can show the reason for that pre-check failure ? Best Regards,Strahil Nikolov В сряда, 10 април 2019 г., 9:05:15 ч. Гринуич-4, Rafi Kavungal Chundattu Parambil <rkavunga(a)redhat.com> написа: Hi Strahil, The name of device is not at all a problem here. Can you please check the log of glusterd, and see if there is any useful information about the failure. Also please provide the output of `lvscan` and `lvs --noheadings -o pool_lv` from all nodes Regards Rafi KC ----- Original Message ----- From: "Strahil Nikolov" <hunter86_bg(a)yahoo.com> To: gluster-users(a)gluster.org Sent: Wednesday, April 10, 2019 2:36:39 AM Subject: [Gluster-users] Gluster snapshot fails Hello Community, I have a problem running a snapshot of a replica 3 arbiter 1 volume. Error: [root@ovirt2 ~]# gluster snapshot create before-423 engine description "Before upgrade of engine from 4.2.2 to 4.2.3" snapshot create: failed: Snapshot is supported only for thin provisioned LV. Ensure that all bricks of engine are thinly provisioned LV. Snapshot command failed Volume info: Volume Name: engine Type: Replicate Volume ID: 30ca1cc2-f2f7-4749-9e2e-cee9d7099ded Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: ovirt1:/gluster_bricks/engine/engine Brick2: ovirt2:/gluster_bricks/engine/engine Brick3: ovirt3:/gluster_bricks/engine/engine (arbiter) Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.enable-shared-storage: enable All bricks are on thin lvm with plenty of space, the only thing that could be causing it is that ovirt1 & ovirt2 are on /dev/gluster_vg_ssd/gluster_lv_engine , while arbiter is on /dev/gluster_vg_sda3/gluster_lv_engine. Is that the issue ? Should I rename my brick's VG ? If so, why there is no mentioning in the documentation ? Best Regards, Strahil Nikolov _______________________________________________ Gluster-users mailing list Gluster-users(a)gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users

1 3

Hosted-engine inaccessible
by Tau Makgaile 12 Apr '19

12 Apr '19

Hi, I am currently experiencing a problem with my Hosted-engine. Hosted-engine disconnected after increasing / partition. The increase went well but after some time the hosted-enigine VM disconnected and has since been giving alerts such as* re-initializingFSM*. Though VMs underneth are running, Hosted-engine --vm-status: *"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail"* There is no backup to restore at the moment. I am looking for a way to bring it up without redeploying the hosted engine. Thanks in advance for your help. Kind regards, Tau

4 5

oVirt Node 4.3 Master on fc28
by femi adegoke 11 Apr '19

11 Apr '19

4.3 Master is looking good!! A few questions: - In Step #2: Fqdns: for a 3 host cluster, are we to add the 2nd & 3rd hosts? - In Step #5: Bricks: What effect does the RAID setting have? Enable Dedupe & Compression is new? Configure LV Cache: what do we enter for SSD? What happens if my disks are already all SSD, do I still benefit from using this?

4 11

Install of oVirt.hosted-engine-setup from galaxy fails
by Vrgotic, Marko 11 Apr '19

11 Apr '19

Dear oVirt team, Install of role oVirt.hosted-engine-setup from Galaxy repo fails: - downloading role 'hosted-engine-setup', owned by oVirt [WARNING]: - oVirt.hosted-engine-setup was NOT installed successfully: - sorry, oVirt.hosted-engine-setup was not found on https://galaxy.ansible.com. Apparently it fails due to name of the repo in the Galaxy which is with “_” instead of expected “-” . Marko Vrgotic

2 1

Re: HostedEngine cleaned up
by Strahil 11 Apr '19

11 Apr '19

The real image is defined within the XML stanza in the vdsm.log when the VM was last started . So if you remember when the last time the HostedEngine was rebooted, you can check the vdsm.log on the host. From there - check the cluster for the file. If it's missing deploy the HostedEngine again (on the previous HostedEngine cluster volume or on a completely new). Then try to import all Storages and then you will be able to import existing VMs. Best Regards, Strahil NikolovOn Apr 11, 2019 20:12, Sakhi Hadebe <sakhi(a)sanren.ac.za> wrote: > > What happened is the engine's root filesystem had filled up. My colleague tried to resize the root lvm. The engine then did not come back. In trying to resolve that he cleaned up the engine and tried to re-install it, no luck in doing that. > > That brought down all the VMs. All VMs are down. we trying to move them into one of the standalone kvm host. We have been trying to locate the VM disk images, with no luck. > > According to the of the VM xml configuration file.the disk file is /rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685 > > Unfortunately we can find it and the solution on teh forum states that we can only find in the associated logical volume, but i think only when teh vm is running. > > The disk images we have been trying to boot up from are the one's we got from the gluster bricks, but the are far small that real images and can't boot > > > On Thu, Apr 11, 2019 at 6:13 PM Simone Tiraboschi <stirabos(a)redhat.com> wrote: >> >> >> >> On Thu, Apr 11, 2019 at 9:46 AM Sakhi Hadebe <sakhi(a)sanren.ac.za> wrote: >>> >>> Hi, >>> >>> We have a situation where the HostedEngine was cleaned up and the VMs are no longer running. Looking at the logs we can see the drive files as: >> >> >> Do you have any guess on what really happened? >> Are you sure that the disks really disappeared? >> >> Please notice that the symlinks under rhev/data-center/mnt/glusterSD/glustermount... are created on the fly only when needed. >> >> Are you sure that your host is correctly connecting the gluster storage domain? >> >>> >>> >>> 2019-03-26T07:42:46.915838Z qemu-kvm: -drive file=/rhev/data-center/mnt/glusterSD/glustermount.goku:_vmstore/9f8ef3f6-53f2-4b02-8a6b-e171b000b420/images/b2b872cd-b468-4f14-ae20-555ed823e84b/76ed4113-51b6-44fd-a3cd-3bd64bf93685,format=qcow2,if=none,id=drive-ua-b2b872cd-b468-4f14-ae20-555ed823e84b,serial=b2b872cd-b468-4f14-ae20-555ed823e84b,werror=stop,rerror=stop,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead >>> >>> I assume this is the disk was writing to before it went down

1 0

Re: Hosted-engine disconnected
by Strahil 11 Apr '19

11 Apr '19

Did you import your storages again after the redeploy ? If yes, there is a tab that allows you to import your templates and VMs. Most probably you will be able to get all of them almost without issues (in my case all VMs got under the default cluster). You only need new glusterfs volume for the redeploy of the engine. Still, have you booted your Engine and connected via VNC to check what happened ? Also, keep in mind that the vdsm.log contains the XML of each VM started on the host. That was how I booted my HostedEngine when the configuration was broken. Create the following alias: alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' With it you will be able to use virsh define XML-file virsh start HostedEngine And so on... If you have an already running HostedEngine (new deploy), try first importing existing storage domains and their VMs. If not, try to power on your HostedEngine with virsh and rescue DVD (for example CentOS 7 install DVD is a good way) and use the troubleshoot menu to fix it. By the way , how did you extend the disks? Best Regards, Strahil NikolovOn Apr 11, 2019 18:33, Tau Makgaile <tau(a)sanren.ac.za> wrote: > > Hi, > > I have been experiencing a problem with my hosted-engine after increasing / partition. The increase went well for few minutes and disconnected. It started by showing bad head and it has since unreachable/ un-ping-able until. It was confusing because vms where still running. I decided to redeploy the engine with a hope that it will pull in the same vms into the dashboard. That did not give any positive results until i opted for a redeployment. things went well until i realized it will need to new glustermounts in order to go through, which also meant it risks losing information around my vms, which i have since stopped in order to allow the redeploment to carry on. > > I now need help to export all of the information around my vms, most importantly the disk images. I have been trying to get some disk images from /gluster_bricks/data/data/ with no luck in booting them up after conversion. I am thinking there might be information missing on this images since they boot into rescue mode. > > Please share more insight on how one can locate the entire disk information or the database where vms were last running. > > I did a check into log files, and did matching of the images with vms names but i have been unsuccessful. > > Thanks in advance for you reply, > > Kind regards, > Tau > >

1 0

Hosted-engine disconnected
by Tau Makgaile 11 Apr '19

11 Apr '19

Hi, I have been experiencing a problem with my hosted-engine after increasing / partition. The increase went well for few minutes and disconnected. It started by showing bad head and it has since unreachable/ un-ping-able until. It was confusing because vms where still running. I decided to redeploy the engine with a hope that it will pull in the same vms into the dashboard. That did not give any positive results until i opted for a redeployment. things went well until i realized it will need to new glustermounts in order to go through, which also meant it risks losing information around my vms, which i have since stopped in order to allow the redeploment to carry on. I now need help to export all of the information around my vms, most importantly the disk images. I have been trying to get some disk images from */gluster_bricks/data/data/ *with no luck in booting them up after conversion. I am thinking there might be information missing on this images since they boot into *rescue mode.* Please share more insight on how one can locate the entire disk information or the database where vms were last running. I did a check into log files, and did matching of the images with vms names but i have been unsuccessful. Thanks in advance for you reply, Kind regards, Tau

1 0

Disk Locked
by Николаев Алексей 11 Apr '19

11 Apr '19

2 2

Expand existing gluster storage in ovirt 4.2/4.3
by adrianquintero＠gmail.com 11 Apr '19

11 Apr '19

Hello, I have a 3 node ovirt 4.3 cluster that I've setup and using gluster (Hyperconverged setup) I need to increase the amount of storage and compute so I added a 4th host (server4.example.com) if it is possible to expand the amount of bricks (storage) in the "data" volume? I did some research and from that research the following caught my eye: old post "https://medium.com/@tumballi/scale-your-gluster-cluster-1-node-at-a-time-62…" So the question is, if taking that approach feasible? , is it even possible an oVirt point of view? --------------------------- My gluster volume: --------------------------- Volume Name: data Type: Replicate Volume ID: 003ffea0-b441-43cb-a38f-ccdf6ffb77f8 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: server1.example1.com:/gluster_bricks/data/data Brick2: server2.example.com:/gluster_bricks/data/data Brick3: server3.example.com:/gluster_bricks/data/data (arbiter) Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: enable performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off Thanks.

3 4

Re: Windows Guest timezone
by Strahil 11 Apr '19

11 Apr '19

I used to set windows machines to use HW clock in UTC , which helped for dual-boot hardware machines. Can't you use that approach and never worry about DST? Best Regards, Strahil NikolovOn Apr 11, 2019 11:42, Stefano Danzi <s.danzi(a)hawai.it> wrote: > > Hello! > > I've a guest windows VM (Server 2016) and hardware clock in oVirt > machine configuration is GMT +1. > Now we are DST period and machine timezone is GMT +2. Engine Dashboard > warn me about timezone > mismatch between configuration and running VM. I can't see a setting to > enable DST on emulated hardware. > > Some ideas? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ENKC37PNVQEZ2…

1 0

Windows Guest timezone
by Stefano Danzi 11 Apr '19

11 Apr '19

Hello! I've a guest windows VM (Server 2016) and hardware clock in oVirt machine configuration is GMT +1. Now we are DST period and machine timezone is GMT +2. Engine Dashboard warn me about timezone mismatch between configuration and running VM. I can't see a setting to enable DST on emulated hardware. Some ideas?

1 0

Ovirt does not support AMD EPYC CPUs?
by Erick Perez 11 Apr '19

11 Apr '19

Hi, In its current release, OVIRT only shows AMD Opteron G1,G2,G3,G4,G5 and no mention of EPYC. This thread mentioned that a respin of Ovirt will address this, but never happened. https://lists.ovirt.org/archives/list/users@ovirt.org/message/2PEAYGK5PE33W… comments?

1 0

Get Disk used size
by mustafa.taha.mu95＠gmail.com 11 Apr '19

11 Apr '19

Hi Can i have way to know the allocated/used size of the VM disk , without install any agent on the VM ?

2 1

change self-hosted engine storage domain
by jason.cox＠harris.com 11 Apr '19

11 Apr '19

Is it possible to change a self-hosted engine's storage domain's settings? I setup a 3-node ovirt + gluster cluster with a dedicated 'engine' storage domain. I can see through the administration portal that the engine storage domain is using a single gluster host and no backup volfile servers. The single gluster host that the engine storage domain settings refer to had to be removed, so now I'm getting issues where VDSM is reporting that it cannot get the volume file information for the engine storage domain. I'm not sure how to go about changing the settings since the only way I've seen how to do this is through the administration portal, which relies on the engine VM (and its storage) being available. What I'd like to do is change the storage domain's path and add mount options for the backup volfile servers.

2 1

Re: Get Disk used size
by Strahil 10 Apr '19

10 Apr '19

What type of storage is being used. If it's file based (NFS,Cluster , other POSIX compliant) you can use qemu-img info to check virtual and actual size of the disk. Still ,this method can be inaccurate. Best Regards, Strahil NikolovOn Apr 10, 2019 15:02, mustafa.taha.mu95(a)gmail.com wrote: > > Hi > Can i have way to know the allocated/used size of the VM disk , without install any agent on the VM ? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IAQPQLKHNZ5QS…

1 0

Re: Different credentials ovirt_vm module
by Ondra Machacek 10 Apr '19

10 Apr '19

Hello, this seems ok to me. Can you please check the engine.log, for some error? On 09/04/2019 22:26, Florian Rädler wrote: > Hi, > > how can i use other credentials (run once) in the ansible playbook for > Sysprep domain join? > > My playbook: > > User_name and root_password does not sound like it’s the right option. > > How can I realize that? > > BR > > Florian > > > ------------------------------------------------------------------------ > > Pflichtangaben anzeigen > <http://www.deutschebahn.com/pflichtangaben/20190408> > > Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie > hier: http://www.deutschebahn.com/de/konzern/datenschutz > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZOKA7M24U5HSZ… >

1 0

accessing ovirt-image-repository from behind proxy?
by Jarosław Prokopowski 10 Apr '19

10 Apr '19

Hi, Is there a way to connect to ovirt-image-repository from ovirt-engine through proxy server? I tried to set http_proxy in /etc/profile of the hosted-engine VM and restarted the VM but it does not help... Errors I get while testing the connection: 2019-04-10 12:36:49,590+03 ERROR [org.ovirt.engine.core.bll.storage.repoimage.GetImagesListQuery] (default task-3) [58f90541-0212-4f1e-9b14-eafc9a8237b9] Failed to retrieve image list: Connec tion timed out (Connection timed out) ... 2019-04-10 12:57:40,609+03 ERROR [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-3) [3c3baa28-c1ac-4783-b362-44b21d7430b5] Command 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' failed: EngineException: (Failed with error PROVIDER_FAILURE and code 5050) Thanks Jarek

1 0

Installing VM - Fedora installer stuck on "Setting up installation source..."
by Jarosław Prokopowski 10 Apr '19

10 Apr '19

Hello, I installed oVirt 4.3.2, added NFS ISO domain and trying to create Fedora 26 VM. The installer starts but is stuck on "Installation Source Setting up installation source..." on the "INSTALLATION SUMMARY" screen. I cannot find any useful information in logs. Do you have any idea what can cause the issue and how to fix that? Thanks Jarek

2 3

Re: Vagrant Plugin
by Jeremy Tourville 10 Apr '19

10 Apr '19

Thanks for your reply Luca, I confirmed the cluster name, it is "Default". I even tried to run the script again and I made sure the D in default was upper case because linux is case sensitive. It still fails in the same way as before. [cid:c5758639-8ce0-46d1-ad0b-900af9cc138e] ________________________________ From: Luca 'remix_tj' Lorenzetto <lorenzetto.luca(a)gmail.com> Sent: Saturday, March 30, 2019 3:34 AM To: Jeremy Tourville Subject: Re: [ovirt-users] Vagrant Plugin Il ven 29 mar 2019, 19:12 Jeremy Tourville <Jeremy_Tourville(a)hotmail.com<mailto:Jeremy_Tourville@hotmail.com>> ha scritto: I am having some trouble getting the Ovirt Vagrant plugin working. I was able to get Vagrant installed and could even run the example scenario listed in the blog. https://www.ovirt.org/blog/2017/02/using-oVirt-vagrant.html My real issue is getting a vm generated by the SecGen project https://github.com/SecGen/SecGen to come up. If I use the VirtualBox provider everything works as expected and I can launch the vm with vagrant up. If I try to run using Ovirt provider it fails. I had originally posted this over in Google groups / Vagrant forums and it was suggested to take it to Ovirt. Hopefully, somebody here has some insights. The process fails quickly with the following output. Can anyone give some suggestions on how to fix the issue? I have also included a copy of my vagrantfile below. Thanks in advance for your assistance! ***Output*** Bringing machine 'escalation' up with 'ovirt4' provider... ==> escalation: Creating VM with the following settings... ==> escalation: -- Name: SecGen-default-scenario-escalation ==> escalation: -- Cluster: default ==> escalation: -- Template: debian_stretch_server_291118 ==> escalation: -- Console Type: spice ==> escalation: -- Memory: ==> escalation: ---- Memory: 512 MB ==> escalation: ---- Maximum: 512 MB ==> escalation: ---- Guaranteed: 512 MB ==> escalation: -- Cpu: ==> escalation: ---- Cores: 1 ==> escalation: ---- Sockets: 1 ==> escalation: ---- Threads: 1 ==> escalation: -- Cloud-Init: false ==> escalation: An error occured. Recovering.. ==> escalation: VM is not created. Please run `vagrant up` first. /home/secgenadmin/.vagrant.d/gems/2.4.4/gems/ovirt-engine-sdk-4.0.12/lib/ovirtsdk4/service.rb:52:in `raise_error': Fault reason is "Operation Failed". Fault detail is "Entity not found: Cluster: name=default". HTTP response code is 404. Hello Jeremy, Looks like you have no cluster called default in your setup. Edit your vagrant file accordingly to your setup. Luca

2 2

Re: Bug 1666795 - Related? - VM's don't start after shutdown on FCP
by Nardus Geldenhuys 10 Apr '19

10 Apr '19

attaching ovirt engine.log

1 1

virt-v2v from Fedora 29 to oVirt 4.3 and ovirtsdk4
by Gianluca Cecchi 10 Apr '19

10 Apr '19

Hello, I'm trying to migrate a VM from a Fedora 29 system to oVirt 4.3.2. I'm trying to use the "-o rhv-upload" option that should be faster. virt-v2v -o rhv-upload -oc https://my_engine/ovirt-engine/api -os my_sd_domain -op /tmp/ovirt-admin-password -of raw -oo rhv-cafile=/tmp/ca.pem -oo rhv-direct --bridge nat:ovn193 my_vm But I get " Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'ovirtsdk4' virt-v2v: error: the Python module ‘ovirtsdk4’ could not be loaded, is it installed? See previous messages for problems. If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] " I see on Fedora 29 repos thease are the only relevant available packages: [root@ope46 ~]# dnf search ovirt ... ovirt-engine-cli.noarch : oVirt Engine Command Line Interface ... ovirt-engine-sdk-java-javadoc.noarch : Javadocs for ovirt-engine-sdk-java ... ovirt-engine-sdk-java.noarch : oVirt Engine Software Development Kit (Java) ... python2-ovirt-engine-sdk.noarch : oVirt Engine Software Development Kit (Python) ... and python2-ovirt-engine-sdk.noarch is related to source ovirt-engine-sdk-python-3.6.9.1-7.fc29.src.rpm so only sdk3 On oVirt repos I see here only for F28: https://resources.ovirt.org/pub/ovirt-4.3/rpm/fc28/x86_64/ trying to install # dnf install https://resources.ovirt.org/pub/ovirt-4.3/rpm/fc28/x86_64/python3-ovirt-eng… gives: Error: Problem: conflicting requests - nothing provides python(abi) = 3.6 needed by python3-ovirt-engine-sdk4-4.3.0-1.fc28.x86_64 (try to add '--skip-broken' to skip uninstallable packages) How can I get correct rpm packages to interact with v4 api of 4.3? Thanks, Gianluca

2 2

Different credentials ovirt_vm module
by Florian Rädler 09 Apr '19

09 Apr '19

Hi, how can i use other credentials (run once) in the ansible playbook for Sysprep domain join? My playbook: [cid:image001.png@01D4EF22.B6352610] User_name and root_password does not sound like it's the right option. How can I realize that? BR Florian ________________________________ Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20190408> Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz

1 0

Cannot get IPMI fencing to work on CIMC
by Alan G 09 Apr '19

09 Apr '19

Hi, I have some Cisco C-220M4 servers with CIMC (not connected to UCS fabric). I can get the fencing to work on the command line like this fence_ipmilan -a XXX -l YYY -p ZZZ --hexadecimal-kg=ABC -o status -P But when I try to configure & test in the Engine UI I get, "Test failed: Internal JSON-RPC error". Obviously the wrong parameters are being passed to the script, but I cannot find a log anywhere which will show me the parameters being used. Anyone else got IPMI working on CIMC? Thanks, Alan

2 1

Re: oVirt 4.3.2 Disk extended broken (UI)
by Strahil 09 Apr '19

09 Apr '19

I guess nobody has experienced such behaviour. I'm pretty sure that initial size of the disk was 100GB and after setting increase by 40 , I see size of 140GB - but my Linux and qemu-img see it as 180GB. Anyone able to guide me through ? Maybe an update to 4.3.3 will solve it? Maybe I should open a bug, but what module is affecting me? Best Regards, Strahil NikolovOn Apr 6, 2019 04:54, Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: > > Hi, > > I have just extended the disk of one of my openSuSE VMs and I have noticed that despite the disk is only 140GiB (in UI), the VM sees it as 180GiB. > > I think that this should not happen at all. > > [root@ovirt1 ee8b1dce-c498-47ef-907f-8f1a6fe4e9be]# qemu-img info c525f67d-92ac-4f36-a0ef-f8db501102fa > image: c525f67d-92ac-4f36-a0ef-f8db501102fa > file format: raw > virtual size: 180G (193273528320 bytes) > disk size: 71G > > Attaching some UI screen shots. > > Note: I have extended the disk via the UI by selecting 40GB (old value in UI -> 100GB). > > Best Regards, > Strahil Nikolov >

1 0

[ANN] oVirt 4.3.3 Third Release Candidate is now available
by Sandro Bonazzola 09 Apr '19

09 Apr '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.3 Third Release Candidate, as of April 9th, 2019. This update is a release candidate of the third in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.6 or later * CentOS Linux (or similar) 7.6 or later * oVirt Node 4.3 (available for x86_64 only) Experimental tech preview for x86_64 and s390x architectures for Fedora 28 is also included. See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available[2] Additional Resources: * Read more about the oVirt 4.3.3 release highlights: http://www.ovirt.org/release/4.3.3/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.3/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig>

1 0

genev_sys_6081 left/entered promiscuous mode
by Arif Ali 09 Apr '19

09 Apr '19

Hi all, I was wondering if anyone seen the messages below in the syslog. I see this set every second, and I am not sure why this is the case, and it's filling the logs quite a lot Mar 22 23:08:08 <snip> kernel: device genev_sys_6081 left promiscuous mode Mar 22 23:08:08 <snip> kernel: i40e 0000:3d:00.0 eno3: port 6081 already offloaded Mar 22 23:08:08 <snip> kernel: i40e 0000:3d:00.1 eno4: port 6081 already offloaded Mar 22 23:08:08 <snip> kernel: i40e 0000:3d:00.2 eno5: port 6081 already offloaded Mar 22 23:08:08 <snip> kernel: i40e 0000:3d:00.3 eno6: port 6081 already offloaded Mar 22 23:08:08 <snip> kernel: device genev_sys_6081 entered promiscuous mode Mar 22 23:08:09 <snip> kernel: i40e 0000:3d:00.0 eno3: UDP port 6081 was not found, not deleting Mar 22 23:08:09 <snip> kernel: i40e 0000:3d:00.1 eno4: UDP port 6081 was not found, not deleting Mar 22 23:08:09 <snip> kernel: i40e 0000:3d:00.2 eno5: UDP port 6081 was not found, not deleting Mar 22 23:08:09 <snip> kernel: i40e 0000:3d:00.3 eno6: UDP port 6081 was not found, not deleting Mar 22 23:08:09 <snip> kernel: device genev_sys_6081 left promiscuous mode -- regards, Arif Ali

2 2

Hosted-engine inaccessible
by tau＠sanren.ac.za 09 Apr '19

09 Apr '19

Hi, I am currently experiencing a problem with my Hosted-engine. Hosted-engine disconnected after increasing / partition. The increase went well but after some time the hosted-enigine VM disconnected and has since been giving alerts such as re-initializingFSM. Though VMs underneth are running, Hosted-engine --vm-status: "reason": "failed liveliness check", "health": "bad", "vm": "up", "detail" There is no backup to restore at the moment. I am looking for a way to bring it up without redeploying the hosted engine. Thanks in advance for your help. Kind regards, Tau

1 0