- Users - oVirt List Archives

vm operation after host fence
by Kapetanakis Giannis 06 Sep '18

06 Sep '18

Hi, Last night we have an incident of a failed host. Engine issued a fence but did not restart the vms running on that node on other operational hosts. I'd like to know if this is normal or I can tune it somehow. Here are some relevant logs from engine: 2018-09-05 03:00:51,496+03 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-827644) [] Host 'v3' is not responding. It will stay in Connecting state for a grace period of 63 seconds and after that an attempt to fence the host will be issued. 2018-09-05 03:01:11,945+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Failed to fetch vms info for host 'v3' - skipping VMs monitoring. 2018-09-05 03:01:48,028+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-827679) [] EVENT_ID: VM_SET_TO_UNKNOWN_STATUS(142), VM vm7 was set to the Unknown status. 2018-09-05 03:02:10,033+03 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (EE-ManagedThreadFactory-engine-Thread-827680) [30369e01] Power-Management: STOP of host 'v3' initiated. 2018-09-05 03:02:55,935+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-827680) [3adcac38] EVENT_ID: VM_WAS_SET_DOWN_DUE_TO_HOST_REBOOT_OR_MANUAL_FENCE(143), Vm vm7 was shut down due to v3 host reboot or manual fence 2018-09-05 03:02:56,018+03 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (EE-ManagedThreadFactory-engine-Thread-827680) [ea0f582] Power-Management: STOP host 'v3' succeeded. 2018-09-05 03:08:20,818+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [326878] EVENT_ID: VDS_DETECTED(13), Status of host v3 was set to Up. 2018-09-05 03:08:23,391+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [] VM '3b1262ef-7fff-40af-b85e-9fd01a4f422b'(vm7) was unexpectedly detected as 'Down' on VDS '4970369d-21c2-467d-9247-c73ca2d71b3e'(v3) (expected on 'null') As you can see, engine does a fence on node v3. vm7 as well as the others running on that node did not re-start. any tips? engine is ovirt-engine-4.2.5.3-1.el7.noarch and host is vdsm-4.20.35-1.el7.x86_64 best regards, Giannis

2 2

oVirt: Training, Webinars, Forums etc!!
by femi adegoke 06 Sep '18

06 Sep '18

Are there any plans for training's/webinars/workshops/documentation etc. Anything to help learn, process more info & get more comfortable with oVirt?

1 0

OVN and impact of missing connectivity with OVN Provider/Central
by Gianluca Cecchi 05 Sep '18

05 Sep '18

Hello, I have VM1 and VM2 with their vnics on OVN. They are running on the same host. Suppose this host (and so its OVN Controller) looses connectivity with the OVN Provider (that in my case runs on oVirt engine, that is an external server). Is it correct/expected that VM1 looses connectivity with VM2 until fixed? So, in other words, is the OVN Provider a sort of single point of failure (eg if I restart enigine in my case)? Thanks, Gianluca

2 2

Engine Setup Error
by Sakhi Hadebe 05 Sep '18

05 Sep '18

Hi, We are deploying the hosted engine on oVirt-Node-4.2.3.1 using the command "hosted-engine --deploy". After providing answers it runs the ansible script and hit the Error when creating glusterfs storage domain. Attached the screenshot of the ERROR. Please help.

5 17

problem with multipath.conf
by g.vasilopoulos＠uoc.gr 05 Sep '18

05 Sep '18

after upgrading to 4.20.39-1.el7 sas multipath stopped being detected I did a diff to the two files Is the current behaviour the correct one or the previous one ? I think sas multipath should also be detected no ? [root@g1-car0136 etc]# diff multipath.conf multipath.conf.201809031555 1c1 < # VDSM REVISION 1.6 --- > # VDSM REVISION 1.5 101,109d100 < } < < # Whitelist FCP and iSCSI devices. < blacklist { < protocol ".*" < } < < blacklist_exceptions { < protocol "(scsi:fcp|scsi:iscsi)"

3 4

ERROR: Malformed metadata for host 3
by info＠linuxfabrik.ch 05 Sep '18

05 Sep '18

Hi, we have a Hyperconverged oVirt Cluster with three machines. All three are running GlusterFS (Replica 3), and on two of them we are running the VMs (because they have better hardware). Now we wanted to add a new oVirt host. Due to some network related errors, adding this host was unsuccessful, so we removed it from GUI and also via "hosted-engine --clean-metadata --host-id=3 --force-clean" (as it was still listed using "hosted-engine --vm-status" after removal from the GUI). Inside the GUI and with "hosted-engine --vm-status", everything looks fine now, the third oVirt host is gone, and there are no problems at all. But in the /var/log/ovirt-hosted-engine-ha/agent.log, we now get every 15 minutes a "MainThread::ERROR::2018-09-05 10:20:17,190::hosted_engine::793::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(process_remote_metadata) Malformed metadata for host 3: received 0 of 512 expected bytes", on both of the remaining oVirt hosts. Can someone explain what is wrong? oVirt 4.2.2.6-1.el7.centos Thank you Markus

1 0

qemu-kvm memory leak in 4.2.5
by Hesham Ahmed 05 Sep '18

05 Sep '18

Starting oVirt 4.2.4 (also in 4.2.5 and maybe in 4.2.3) I am facing some sort of memory leak. The memory usage on the hosts keep increasing till it reaches somewhere around 97%. Putting the host in maintenance and back resolves it. The memory usage by the qemu-kvm processes is way above the defined VM memory for instance below is the memory usage of a VM: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12271 qemu 20 0 35.4g 30.9g 8144 S 13.3 49.3 9433:15 qemu-kvm The VM memory settings are: Defined Memory: 8192 MB Physical Memory Guaranteed: 5461 MB This is for a 3 node hyperconverged cluster running on latest oVirt Node 4.2.5.

3 2

Local storage formatting
by Matt Simonsen 04 Sep '18

04 Sep '18

Hello, I'm running oVirt with several data centers, some with NFS storage and some with local storage. I had problems in the past with a large pool and local storage. The problem was nodectl showed the pool being too full (I think >80%), but it was only the images that made the pool "full" -- and this storage was carefully setup such that there was no chance it would actually fill. The LVs for oVirt itself were all under 20%, yet nodectl still reported the pool was too full. My solution so far has been to use our RAID card tools, so that sda is the oVirt node install, and sdb is for images. There are probably other good reasons for me to handle it this way, for example being able to use different RAID levels, but I'm hoping someone can confirm my partitioning below doesn't have some risk I'm now yet aware of. I setup a new volume group for images, as below: [root@node4-g8-h4 multipath]# pvs PV VG Fmt Attr PSize PFree /dev/mapper/3600508b1001c7e172160824d7b204c3b2 onn_node4-g8-h4 lvm2 a-- <119.00g <22.85g /dev/sdb1 data lvm2 a-- 1.13t <361.30g [root@node4-g8-h4 multipath]# vgs VG #PV #LV #SN Attr VSize VFree data 1 1 0 wz--n- 1.13t <361.30g onn_node4-g8-h4 1 13 0 wz--n- <119.00g <22.85g [root@node4-g8-h4 multipath]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert images_main data -wi-ao---- 800.00g home onn_node4-g8-h4 Vwi-aotz-- 1.00g pool00 4.79 ovirt-node-ng-4.2.5.1-0.20180816.0 onn_node4-g8-h4 Vwi---tz-k 64.10g pool00 root ovirt-node-ng-4.2.5.1-0.20180816.0+1 onn_node4-g8-h4 Vwi---tz-- 64.10g pool00 ovirt-node-ng-4.2.5.1-0.20180816.0 ovirt-node-ng-4.2.6-0.20180903.0 onn_node4-g8-h4 Vri---tz-k 64.10g pool00 ovirt-node-ng-4.2.6-0.20180903.0+1 onn_node4-g8-h4 Vwi-aotz-- 64.10g pool00 ovirt-node-ng-4.2.6-0.20180903.0 4.83 pool00 onn_node4-g8-h4 twi-aotz-- 91.10g 8.94 0.49 root onn_node4-g8-h4 Vwi---tz-- 64.10g pool00 swap onn_node4-g8-h4 -wi-ao---- 4.00g tmp onn_node4-g8-h4 Vwi-aotz-- 1.00g pool00 4.87 var onn_node4-g8-h4 Vwi-aotz-- 15.00g pool00 3.31 var_crash onn_node4-g8-h4 Vwi-aotz-- 10.00g pool00 2.86 var_log onn_node4-g8-h4 Vwi-aotz-- 8.00g pool00 3.57 var_log_audit onn_node4-g8-h4 Vwi-aotz-- 2.00g pool00 4.89 The images_main is setup as "Block device for filesystems" with ext4. Is there any reason I should consider pool for thinly provisioned volumes? I don't need to over-allocate storage and it seems to me like a fixed partition is ideal. Please confirm or let me know if there's anything else I should consider. Thanks Matt

2 3

self-hosted engine lost password
by Rob Epping 04 Sep '18

04 Sep '18

Hello list, I'm trying to update our ovirt self-hosted engine to the latest version using the docs at https://www.ovirt.org/documentation/upgrade-guide/chap-Updates_between_ Minor_Releases/#updating-the-ovirt-self-hosted-engine-and-underlying- virtualization-hosts For this I need to logon to the engine VM, but none of the passwords we used seems to work. I can not seem to find information on how te reset passwords of the hosted engine, so I'm stuck. Am I doing something wrong or is access to the self-hosted engine required. When access is required, is there a way to reset the password? THNX && GRTNX, RobJE

2 2

Upgraded host, engine now won't boot
by Jim Kusznir 04 Sep '18

04 Sep '18

Hello: I saw that there were updates to my ovirt-4.2 3 node hyperconverged system, so I proceeded to apply them the usual way through the UI. At one point, the hosted engine was migrated to one of the upgraded hosts, and then went "unstable" on me. Now, the hosted engine appears to be crashed: It gets powered up, but it never boots up to the point where it responds to pings or allows logins. After a while, the hosted engine shows status (via console "hosted-engine --vm-status" command) "Powering Down". It stays there for a long time. I tried forcing a poweroff then powering it on, but again, it never gets up to where it will respond to pings. --vm-status shows bad health, but up. I tried running the hosted-engine --console command, but got: [root@ovirt1 ~]# hosted-engine --console The engine VM is running on this host Connected to domain HostedEngine Escape character is ^] error: internal error: cannot find character device <null> [root@ovirt1 ~]# I tried to run the hosted-engine --upgrade-appliance command, but it hangs at obtaining certificate (understandably, as the hosted-engine is not up). How do i recover from this? And what caused this? --Jim

2 10

ovirt4 api search storage domain
by David David 04 Sep '18

04 Sep '18

hi need to lookup storage name by vm disk id. how to do it in python or in java? thanks

2 1

Wrong network threshold limit warnings on 4.2.5
by Florian Schmid 04 Sep '18

04 Sep '18

Good morning, since we have upgraded to version 4.2.5, we get a lot of warnings about network interface exceeded defined threshold limits. For example: Aug 31, 2018, 7:54:05 AM Host xxx has network interface which exceeded the defined threshold [95%] (enp9s0.80: transmit rate[100%], receive rate [12%]) This is a 10 Gbit interface and on our monitoring software, which is getting network statistics every 10s, the bandwidth of TX was 150 Mbit maximum at this time, so far away from being 100%. Could it be, that the engine detected the wrong interface speed or there is a calculation error? In the engine for this host, I have 10000 Mbps for all interfaces. I have checked now all those warnings on our different hosts and they happen every time, we go over 100 Mbit and this is for sure quite often... Can I maybe disable these warnings, because we have it anyway in our monitoring software? If you need any logs, please ask. BR Florian Schmid

5 10

Bricks do not appear in oVirt Manager UI under Hosts unless manually created on nodes.
by payden.pringle+ovirtlist＠gmail.com 03 Sep '18

03 Sep '18

I have a CentOS 7.5 system running the oVirt-Engine package as the manager for three oVirt Nodes (version 4.2). Two of the nodes have a 200GB disk and one has a 2GB disk. I am attempting to create a Gluster Volume that replicates between the two 200GB disks with the 2GB as Arbiter. When I create the bricks by going to Hosts > host1 > Storage Devices > Create Brick (with the 200GB device selected), it succeeds, but no bricks appear under the Bricks tab for the Host. Doing the same for host2 and host3 results in the same problem. If I run the following command on a host `gluster volume create test-volume replica 3 arbiter 1 192.168.5.{50,60,40}:/gluster-bricks/200GBbrick/test`, it creates the volume correctly and the bricks then appear as they should. However, then these Bricks are labeled as "Down" and when I go to Storage > Volumes, I get this error: ``` Uncaught exception occurred. Please try reloading the page. Details: (TypeError) : Cannot read property 'a' of null Please have your administrator check the UI logs ``` The Volume is also shown as "Down". SELinux is enabled on my CentOS 7.5 system. I'm not really sure what the correct method is to create bricks then a Volume in Gluster through the oVirt Management UI, but I've followed the guides and this is where it has got me.

2 1

oVirt Node 4.2.3.1. to 4.2.5 upgrade trouble, log attached
by Matt Simonsen 03 Sep '18

03 Sep '18

Hello all, I've emailed about similar trouble with an oVirt Node upgrade using the ISO install. I've attached the /tmp/imgbased.log file in hopes it will help give a clue as to why the trouble. Since these use NFS storage I can rebuild, but would like to know, ideally, what caused the upgrade to break. Truthfully following the install, I don't think I have done *that* much to these systems, so I'm not sure what would have caused the problem. I have done several successful upgrades in the past and most of my standalone systems have been working great. I've been really happy with oVirt, so kudos to the team. Thanks for any help, Matt

5 13

the issue of starting a VM
by hhz711＠126.com 03 Sep '18

03 Sep '18

hello: I created a VM on node1, but when I power on the vm and try to boot from the CD-ROM with ISO, but it's stucked . the boot window only show below info: "seaBIOS ( version xxxxxx） Machine UUID xxxxxxxx " Did you guys have met this before ?

1 0

Gluster clients intermittently hang until first gluster server in a Replica 1 Arbiter 1 cluster is rebooted, server error: 0-management: Unlocking failed & client error: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2131 sent = <datestamp>. timeout = 1800
by Sam McLeod 03 Sep '18

03 Sep '18

We've got an odd problem where clients are blocked from writing to Gluster volumes until the first node of the Gluster cluster is rebooted. I suspect I've either configured something incorrectly with the arbiter / replica configuration of the volumes, or there is some sort of bug in the gluster client-server connection that we're triggering. I was wondering if anyone has seen this or could point me in the right direction? Environment: Typology: 3 node cluster, replica 2, arbiter 1 (third node is metadata only). Version: Client and Servers both running 4.1.3, both on CentOS 7, kernel 4.18.x, (Xen) VMs with relatively fast networked SSD storage backing them, XFS. Client: Native Gluster FUSE client mounting via the kubernetes provider Problem: Seemingly randomly some clients will be blocked / are unable to write to what should be a highly available gluster volume. The client gluster logs show it failing to do new file operations across various volumes and all three nodes of the gluster. The server gluster (or OS) logs do not show any warnings or errors. The client recovers and is able to write to volumes again after the first node of the gluster cluster is rebooted. Until the first node of the gluster cluster is rebooted, the client fails to write to the volume that is (or should be) available on the second node (a replica) and third node (an arbiter only node). What 'fixes' the issue: Although the clients (kubernetes hosts) connect to all 3 nodes of the Gluster cluster - restarting the first gluster node always unblocks the IO and allows the client to continue writing. Stopping and starting the glusterd service on the gluster server is not enough to fix the issue, nor is restarting its networking. This suggests to me that the volume is unavailable for writing for some reason and restarting the first node in the cluster either clears some sort of TCP sessions between the client-server or between the server-server replication. Expected behaviour: If the first gluster node / server had failed or was blocked from performing operations for some reason (which it doesn't seem it is), I'd expect the clients to access data from the second gluster node and write metadata to the third gluster node as well as it's an arbiter / metadata only node. If for some reason the a gluster node was not able to serve connections to clients, I'd expect to see errors in the volume, glusterd or brick log files (there are none on the first gluster node). If the first gluster node was for some reason blocking IO on a volume, I'd expect that node either to show as unhealthy or unavailable in the gluster peer status or gluster volume status. Client gluster errors: staging_static in this example is a volume name. You can see the client trying to connect to the second and third nodes of the gluster cluster and failing (unsure as to why?) The server side logs on the first gluster node do not show any errors or problems, but the second / third node show errors in the glusterd.log when trying to 'unlock' the 0-management volume on the first node. On a gluster client (a kubernetes host using the kubernetes connector which uses the native fuse client) when its blocked from writing but the gluster appears healthy (other than the errors mentioned later): [2018-09-02 15:33:22.750874] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1cce sent = 2018-09-02 15:03:22.417773. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 15:33:22.750989] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 16:03:23.097905] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2e21 sent = 2018-09-02 15:33:22.765751. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 16:03:23.097988] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 16:33:23.439172] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1d4b sent = 2018-09-02 16:03:23.098133. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 16:33:23.439282] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 17:03:23.786858] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2ee7 sent = 2018-09-02 16:33:23.455171. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 17:03:23.786971] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 17:33:24.160607] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1dc8 sent = 2018-09-02 17:03:23.787120. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 17:33:24.160720] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 18:03:24.505092] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2faf sent = 2018-09-02 17:33:24.173153. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 18:03:24.505185] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 18:33:24.841248] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1e45 sent = 2018-09-02 18:03:24.505328. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 18:33:24.841311] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 19:03:25.204711] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3074 sent = 2018-09-02 18:33:24.855372. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 19:03:25.204784] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 19:33:25.533545] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1ec2 sent = 2018-09-02 19:03:25.204977. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 19:33:25.533611] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 20:03:25.877020] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3138 sent = 2018-09-02 19:33:25.545921. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 20:03:25.877098] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 20:33:26.217858] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1f3e sent = 2018-09-02 20:03:25.877264. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 20:33:26.217973] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 21:03:26.588237] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x31ff sent = 2018-09-02 20:33:26.233010. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 21:03:26.588316] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 21:33:26.912334] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1fbb sent = 2018-09-02 21:03:26.588456. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 21:33:26.912449] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 22:03:37.258915] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x32c5 sent = 2018-09-02 21:33:32.091009. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 22:03:37.259000] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 22:33:37.615497] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2039 sent = 2018-09-02 22:03:37.259147. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 22:33:37.615574] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 23:03:37.940969] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3386 sent = 2018-09-02 22:33:37.629655. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 23:03:37.941049] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 23:33:38.270998] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x20b5 sent = 2018-09-02 23:03:37.941199. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 23:33:38.271078] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-03 00:03:38.607186] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3447 sent = 2018-09-02 23:33:38.285899. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-03 00:03:38.607263] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-03 00:33:38.934385] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2131 sent = 2018-09-03 00:03:38.607410. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-03 00:33:38.934479] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-03 01:03:39.256842] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x350c sent = 2018-09-03 00:33:38.948570. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-03 01:03:39.256972] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-03 01:33:39.614402] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x21ae sent = 2018-09-03 01:03:39.258166. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-03 01:33:39.614483] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] On the second gluster server: We are seeing the following error in the glusterd.log file when the client is blocked from writing the volume, I think this is probably the most important information about the error and suggests a problem with the first node but doesn't explain the client behaviour: [2018-09-02 08:31:03.902272] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on <FQDN of the first gluster node>. Please check log file for details. [2018-09-02 08:31:03.902477] E [MSGID: 106151] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s) Note in the above error: 1. I'm not sure which log to check (there doesn't seem to be a management brick / brick log)? 2. If there's a problem with the first node, why isn't it rejected from the gluster / taken offline / the health of the peers or volume list degraded? 3. Why does the client fail to write to the volume rather than (I'm assuming) trying the second (or third I guess) node to write to the volume? We are also seeing the following errors repeated a lot in the logs, both when the volumes are working and when there's an issue in the brick log (/var/log/glusterfs/bricks/mnt-gluster-storage-staging_static.log): [2018-09-03 01:58:35.128923] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.128957] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3d60, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.128983] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.129016] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3e2a, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.129042] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.129077] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3ef6, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.129149] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.129191] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3fc6, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.129219] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed Gluster volume information: # gluster volume info staging_static Volume Name: staging_static Type: Replicate Volume ID: 7f3b8e91-afea-4fc6-be83-3399a089b6f3 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: <first gluster node.fqdn>:/mnt/gluster-storage/staging_static Brick2: <second gluster node.fqdn>:/mnt/gluster-storage/staging_static Brick3: <third gluster node.fqdn>:/mnt/gluster-storage/staging_static (arbiter) Options Reconfigured: storage.fips-mode-rchecksum: true cluster.self-heal-window-size: 16 cluster.shd-wait-qlength: 4096 cluster.shd-max-threads: 8 performance.cache-min-file-size: 2KB performance.rda-cache-limit: 1GB network.inode-lru-limit: 50000 server.outstanding-rpc-limit: 256 transport.listen-backlog: 2048 performance.write-behind-window-size: 512MB performance.stat-prefetch: true performance.io-thread-count: 16 performance.client-io-threads: true performance.cache-size: 1GB performance.cache-refresh-timeout: 60 performance.cache-invalidation: true cluster.use-compound-fops: true cluster.readdir-optimize: true cluster.lookup-optimize: true cluster.favorite-child-policy: size cluster.eager-lock: true client.event-threads: 4 nfs.disable: on transport.address-family: inet diagnostics.brick-log-level: ERROR diagnostics.client-log-level: ERROR features.cache-invalidation-timeout: 300 features.cache-invalidation: true network.ping-timeout: 15 performance.cache-max-file-size: 3MB performance.md-cache-timeout: 300 server.event-threads: 4 Thanks in advance, -- Sam McLeod (protoporpoise on IRC) https://smcleod.net https://twitter.com/s_mcleod Words are my own opinions and do not necessarily represent those of my employer or partners.

1 1

Windows vm and cpu type
by carl langlois 31 Aug '18

31 Aug '18

Hi all, I am trying to create a windows VM on my cluster that is set with cpu type to Nehalem. The probleme is that ovirt tell me that the guest os does not support this cpu.. What is the best ways to fix it without impacting my 41 running vms?. Thanks Regards

2 3

Re: ovirt-engine-extension-aaa-ldap-setup credentials invalid
by Douglas Duckworth 31 Aug '18

31 Aug '18

Thanks! I set that after completing setup though still cannot login: 2018-08-30 10:42:46,470-04 ERROR [org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default task-5) [] Cannot authenticate user 'me@scu' connecting from '10.0.0.136': Unable to log in. Verify your login information or contact the system administrator. How can I get get more verbose logging? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Thu, Aug 30, 2018 at 8:36 AM, Ondra Machacek <omachace(a)redhat.com> wrote: > On 08/29/2018 08:04 PM, Douglas Duckworth wrote: > > Hi > > > > I am not able to finish ovirt-engine-extension-aaa-ldap-setup due to the > > error "credentials invalid." > > > > My password's correct. Our directory server's OpenLDAP standard schema. > > Our accounts are under OU=People. We are binding with a service account > > that has access to the entire DN. > > > > Any way to discern more information from attached logs? > > As you can see you get: > > 2018-08-29 13:47:44,129-04 WARNING Exception: anonymous bind disallowed > > More info how to solve this issue is here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. > ovirt.org_pipermail_users_2018-2DFebruary_086924.html&d=DwICaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s= > GKdSUL68LUZfQo0jXz6oizzkl38dkjJBqsxlNjlNhg0&e= > > > > > > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York Avenue > > New York, NY 10065 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://urldefense.proofpoint. > com/v2/url?u=https-3A__www.ovirt.org_site_privacy-2Dpolicy_&d=DwICaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s= > EMrnI4emKvoDHpfvwPDhoIiX_UZd3_cUeYWCl26APHk&e= > > oVirt Code of Conduct: https://urldefense.proofpoint. > com/v2/url?u=https-3A__www.ovirt.org_community_about_ > community-2Dguidelines_&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_ > P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s=K5tHvffPexNoF9tOFEFopJcv4ROA1s > JUfLr9dvAglXM&e= > > List Archives: https://urldefense.proofpoint. > com/v2/url?u=https-3A__lists.ovirt.org_archives_list_users- > 40ovirt.org_message_ROWSZGMEMSYKUWEIDAKZMSVTKHPPOEDN_&d=DwICaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s= > lyUWPBTItHvb0nXgveWBdPUGNTyuAdUx4OnaME4dOrY&e= > > >

2 1

Disk Statistics error
by Hari Prasanth Loganathan 31 Aug '18

31 Aug '18

Hi Team, I try to get the disk statistics using the following *Url: * * /ovirt-engine/api/disks?follow=statistics* and I get the following error in oVirt, 2018-08-17 15:34:24,663+08 ERROR [io.undertow.request] (default task-815) UT005023: Exception handling request to /ovirt-engine/api/v4/disks: java.lang.RuntimeException: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: Problem following 'statistics' link in Disks entity. at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl( RequestDispatcherImpl.java:245) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup( RequestDispatcherImpl.java:147) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward( RequestDispatcherImpl.java:111) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.api.restapi.invocation.VersionFilter. doFilter(VersionFilter.java:180) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.VersionFilter. doFilter(VersionFilter.java:98) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.api.restapi.invocation.CurrentFilter. doFilter(CurrentFilter.java:117) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.CurrentFilter. doFilter(CurrentFilter.java:72) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter. doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter. doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter. doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter. doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter. doFilter(SessionValidationFilter.java:59) [aaa.jar:] Caused by: java.lang.IllegalStateException: Problem following 'statistics' link in Disks entity. at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. fetchData(LinkFollower.java:185) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. followLink(LinkFollower.java:144) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. followLinks(LinkFollower.java:132) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. followLinks(LinkFollower.java:53) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BaseBackendResource. follow(BaseBackendResource.java:412) [restapi-jaxrs.jar:] at org.ovirt.engine.api.resource.DisksResource.doList(DisksResource.java:222) [restapi-definition.jar:] at sun.reflect.GeneratedMethodAccessor407.invoke(Unknown Source) [:1.8.0_181] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker. internalInvokeOnTarget(ResourceMethodInvoker.java:510) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker. invokeOnTargetAfterFilter(ResourceMethodInvoker.java:401) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$ invokeOnTarget$0(ResourceMethodInvoker.java:365) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.interception. PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget( ResourceMethodInvoker.java:367) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( ResourceMethodInvoker.java:339) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker. invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke( ResourceLocatorInvoker.java:100) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke( SynchronousDispatcher.java:441) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] ... 96 more Caused by: java.lang.IllegalStateException: Problem fetching 'statistics' from Disk at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. fetchData(LinkFollower.java:219) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. fetchData(LinkFollower.java:182) [restapi-jaxrs.jar:] ... 114 more Caused by: java.lang.NullPointerException Any thoughts on this? Thanks, Hari -- DISCLAIMER - *MSysTechnologies LLC* This email message, contents and its attachments may contain confidential, proprietary or legally privileged information and is intended solely for the use of the individual or entity to whom it is actually intended. If you have erroneously received this message, please permanently delete it immediately and notify the sender. If you are not the intended recipient of the email message,you are notified strictly not to disseminate,distribute or copy this e-mail.E-mail transmission cannot be guaranteed to be secure or error-free as Information could be intercepted, corrupted, lost, destroyed, incomplete or contain viruses and MSysTechnologies LLC accepts no liability for the contents and integrity of this mail or for any damage caused by the limitations of the e-mail transmission.

2 6

Possible vdsmd issues - " WARN Worker blocked"
by Johnson, Tim 31 Aug '18

31 Aug '18

Hi all, We have had a bit of a strange issue with the starting of the self-hosted engine due to the vdsm and ovirt-ha processes having errors and not able to be started. Running Ovirt version 4.2 with Gluster mounts (not hyper converged) gluster 3.12.13 replicated volumes with arbiter brick. The mounts for the storage are mounted on all the Ovirt hosts and able to read/touch files on the mounts. Unfortunately we can not pin a specific timeframe when this started happening. Meaning we don’t know if it started after a specific update or any other event. The vdsmd is throwing errors (bottom – with version numbers) when trying to check the status of the self -hosted engine getting : # hosted-engine --vm-status The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. Thanks for any assistance. systemctl status ovirt-ha-agent -l ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Thu 2018-08-30 13:24:40 CDT; 2s ago Process: 17135 ExecStart=/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent (code=exited, status=157) Main PID: 17135 (code=exited, status=157) systemctl status ovirt-ha-broker -l ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2018-08-30 13:10:35 CDT; 14min ago Main PID: 14863 (ovirt-ha-broker) Tasks: 4 CGroup: /system.slice/ovirt-ha-broker.service └─14863 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker Aug 30 13:10:35 rrb-vmhost1.rrc.local systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Aug 30 13:10:35 rrb-vmhost1.rrc.local systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker... Aug 30 13:14:44 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:14:44 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.Action.start_domain_monitor ERROR Error in RPC call: Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.Action.start_domain_monitor ERROR Error in RPC call: Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Failed to update state. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 82, in run if (self._status_broker._inquire_whiteboard_lock() or File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 190, in _inquire_whiteboard_lock self.host_id, self._lease_file) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 128, in host_id raise ex.HostIdNotLockedError("Host id is not set") HostIdNotLockedError: Host id is not set Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Trying to restart the broker Aug 30 13:19:42 rrb-vmhost1.rrc.local python[14863]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Aug 30 13:19:48 rrb-vmhost1.rrc.local python[14863]: communication with ABRT daemon failed: timed out glusterfs-api-3.12.13-1.el7.x86_64 glusterfs-server-3.12.13-1.el7.x86_64 glusterfs-fuse-3.12.13-1.el7.x86_64 vdsm-gluster-4.20.35-1.el7.x86_64 centos-release-gluster312-1.0-2.el7.centos.noarch glusterfs-client-xlators-3.12.13-1.el7.x86_64 glusterfs-geo-replication-3.12.13-1.el7.x86_64 glusterfs-libs-3.12.13-1.el7.x86_64 glusterfs-cli-3.12.13-1.el7.x86_64 glusterfs-rdma-3.12.13-1.el7.x86_64 glusterfs-events-3.12.13-1.el7.x86_64 glusterfs-3.12.13-1.el7.x86_64 libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64 python2-gluster-3.12.13-1.el7.x86_64 glusterfs-gnfs-3.12.13-1.el7.x86_64 ovirt-setup-lib-1.1.4-1.el7.centos.noarch ovirt-iso-uploader-4.2.0-1.el7.centos.noarch ovirt-engine-setup-base-4.2.5.3-1.el7.noarch ovirt-host-dependencies-4.2.3-1.el7.x86_64 ovirt-guest-agent-common-1.0.14-1.el7.noarch python-ovirt-engine-sdk4-4.2.7-2.el7.x86_64 ovirt-engine-setup-plugin-ovirt-engine-common-4.2.5.3-1.el7.noarch ovirt-vmconsole-1.0.5-4.el7.centos.noarch ovirt-engine-tools-backup-4.2.5.3-1.el7.noarch ovirt-guest-agent-windows-1.0.14-1.el7.centos.noarch cockpit-ovirt-dashboard-0.11.31-1.el7.noarch ovirt-host-deploy-1.7.4-1.el7.noarch ovirt-engine-lib-4.2.5.3-1.el7.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-hosted-engine-ha-2.2.16-1.el7.noarch ovirt-guest-tools-iso-4.2-1.el7.centos.noarch ovirt-imageio-common-1.4.2-0.el7.noarch ovirt-hosted-engine-setup-2.2.25-1.el7.noarch ovirt-imageio-daemon-1.4.2-0.el7.noarch ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch ovirt-provider-ovn-driver-1.2.13-1.el7.noarch ovirt-host-4.2.3-1.el7.x86_64 vdsm-jsonrpc-4.20.35-1.el7.noarch vdsm-hook-ethtool-options-4.20.35-1.el7.noarch vdsm-gluster-4.20.35-1.el7.x86_64 vdsm-api-4.20.35-1.el7.noarch vdsm-yajsonrpc-4.20.35-1.el7.noarch vdsm-client-4.20.35-1.el7.noarch vdsm-hook-fcoe-4.20.35-1.el7.noarch vdsm-network-4.20.35-1.el7.x86_64 vdsm-hook-vmfex-dev-4.20.35-1.el7.noarch vdsm-hook-vhostmd-4.20.35-1.el7.noarch vdsm-http-4.20.35-1.el7.noarch vdsm-hook-openstacknet-4.20.35-1.el7.noarch vdsm-python-4.20.35-1.el7.noarch vdsm-hook-vfio-mdev-4.20.35-1.el7.noarch vdsm-common-4.20.35-1.el7.noarch vdsm-4.20.35-1.el7.x86_64 Aug 30 13:02:42 rrb-vmhost1.rrc.local vdsm[6792]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f713c0b0fd0> at 0x7f713c063050> timeout=15, duration=75 at 0x7f7170698c90> task#=354 at 0x7f713c10ec90>, traceback: File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run self._execute_task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in __call__ self._func() File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in __call__ stats = hostapi.get_stats(self._cif, self._samples.stats()) File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats ret['haStats'] = _getHaInfo() File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo stats = instance.get_all_stats() File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats stats = broker.get_stats_from_storage() File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage result = self._proxy.get_stats() File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request verbose=self.__verbose File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request response = h.getresponse(buffering=True) File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse response.begin() File: "/usr/lib64/python2.7/httplib.py", line 444, in begin version, status, reason = self._read_status() File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status line = self.fp.readline(_MAXLINE + 1) File: "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize)

2 1

Hosted Engine, Node, Local Storage, All in One
by Jeremy Tourville 30 Aug '18

30 Aug '18

Can anyone tell me if this can be done? I have read various articles and I'm not sure. http://blog.domb.net/?p=2141 https://www.ovirt.org/develop/release-management/features/integration/allin…<http://blog.domb.net/?p=2141> According to my google search results the ovirt article date is within the last year approximately but I says deprecated after version 3.6 yet the 1st article makes it seem it can be done. I know this goes against standard practice but there is a reason I'm trying to do it. If it is possible, can anyone point me to a guide or give me an overview of the steps needed? Thank you for your help!

2 1

Re: Turn Off Email Alerts
by Douglas Duckworth 30 Aug '18

30 Aug '18

Fixed *THANK YOU!* Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Wed, Aug 29, 2018 at 6:09 PM, Martin Sivak <msivak(a)redhat.com> wrote: > Hi, > > > Yes, indeed! > > > > How can I change this internal Python setting? > > The same as the notification regex. Using hosted-engine > --set-shared-config <key> <value> --type=<type> > > Executing hosted-engine --get-shared-config xxx (or anything else that > does not exist) should give you the list of all types and keys you can > change (on 4.2 fo sure and 4.1 very probably too). > > Best regards > > Martin Sivak > > > On Wed, Aug 29, 2018 at 11:56 PM, Douglas Duckworth > <dod2014(a)med.cornell.edu> wrote: > > Yes, indeed! > > > > How can I change this internal Python setting? > > > > On Wed, Aug 29, 2018, 5:43 PM Martin Sivak <msivak(a)redhat.com> wrote: > >> > >> Hi, > >> > >> two clarifications: > >> > >> Hosted engine is sending those emails using built-in Python SMTP > >> client that talks directly yo the SMTP server specified during install > >> time. We default to localhost, but you might have changed it. > >> > >> > notify.state_transition : maintenance|start|stop|migrate|up|down, > type : > >> > broker > >> > >> The value here is a regular expression that is matched against the > >> state transition string in the email. > >> > >> Best regards > >> > >> Martin Sivak > >> > >> On Wed, Aug 29, 2018 at 10:34 PM, Douglas Duckworth > >> <dod2014(a)med.cornell.edu> wrote: > >> > Thanks for sharing > >> > > >> > I may want to do that > >> > > >> > Though first I want to understand how the emails are arriving. > >> > > >> > I stooped ovirt-engine-notifier.service and postfix.service on all > hosts > >> > and > >> > the hosted engine. So how are the email being delivered? They are > not > >> > running senmail so I don't understand what daemon's sending these > >> > messages. > >> > > >> > Thanks, > >> > > >> > Douglas Duckworth, MSc, LFCS > >> > HPC System Administrator > >> > Scientific Computing Unit > >> > Weill Cornell Medicine > >> > 1300 York Avenue > >> > New York, NY 10065 > >> > E: doug(a)med.cornell.edu > >> > O: 212-746-6305 > >> > F: 212-746-8690 > >> > > >> > > >> > > >> > On Wed, Aug 29, 2018 at 3:16 PM, Simone Tiraboschi < > stirabos(a)redhat.com> > >> > wrote: > >> >> > >> >> Hi, > >> >> you can change the list of status you want to be notified about with > >> >> hosted-engine --set-shared-config notify.state_transition > >> >> The default is: > >> >> > >> >> [root@hehost01 ~]# hosted-engine --get-shared-config > >> >> notify.state_transition --type=broker > >> >> > >> >> notify.state_transition : maintenance|start|stop|migrate|up|down, > type > >> >> : > >> >> broker > >> >> > >> >> > >> >> On Wed, Aug 29, 2018 at 7:47 PM Douglas Duckworth > >> >> <dod2014(a)med.cornell.edu> wrote: > >> >>> > >> >>> I agree however we are in testing phase so I make changes a lot. > >> >>> Therefore alerts are not presently needed. > >> >>> > >> >>> So how to I turn them off? > >> >>> > >> >>> These steps do not work on hosted engine: > >> >>> > >> >>> > >> >>> me@ovirt-engine[~]$ sudo systemctl stop postfix.service > >> >>> me@ovirt-engine[~]$ sudo systemctl stop > ovirt-engine-notifier.service > >> >>> me@ovirt-engine[~]$ sudo systemctl status > >> >>> ovirt-engine-notifier.service > >> >>> ● ovirt-engine-notifier.service - oVirt Engine Notifier > >> >>> Loaded: loaded > >> >>> (/usr/lib/systemd/system/ovirt-engine-notifier.service; > >> >>> enabled; vendor preset: disabled) > >> >>> Active: inactive (dead) since Wed 2018-08-29 13:41:55 EDT; 3s ago > >> >>> Process: 1814 > >> >>> > >> >>> ExecStart=/usr/share/ovirt-engine/services/ovirt-engine- > notifier/ovirt-engine-notifier.py > >> >>> --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, > >> >>> status=0/SUCCESS) > >> >>> Main PID: 1814 (code=exited, status=0/SUCCESS) > >> >>> > >> >>> Aug 25 12:09:31 ovirt-engine.pbtech systemd[1]: Starting oVirt > Engine > >> >>> Notifier... > >> >>> Aug 25 12:09:33 ovirt-engine.pbtech systemd[1]: Started oVirt Engine > >> >>> Notifier. > >> >>> Aug 29 13:41:54 ovirt-engine.pbtech systemd[1]: Stopping oVirt > Engine > >> >>> Notifier... > >> >>> Aug 29 13:41:55 ovirt-engine.pbtech systemd[1]: Stopped oVirt Engine > >> >>> Notifier. > >> >>> > >> >>> I still get an email every time I put a host in maint mode for > >> >>> example. > >> >>> > >> >>> > >> >>> Thanks, > >> >>> > >> >>> Douglas Duckworth, MSc, LFCS > >> >>> HPC System Administrator > >> >>> Scientific Computing Unit > >> >>> Weill Cornell Medicine > >> >>> 1300 York Avenue > >> >>> New York, NY 10065 > >> >>> E: doug(a)med.cornell.edu > >> >>> O: 212-746-6305 > >> >>> F: 212-746-8690 > >> >>> > >> >>> > >> >>> > >> >>> On Tue, Aug 28, 2018 at 4:43 PM, Johan Bernhardsson <johan(a)kafit.se > > > >> >>> wrote: > >> >>>> > >> >>>> Those alerts are also coming from hosted-engine that keeps ovirt > >> >>>> manager > >> >>>> running. > >> >>>> > >> >>>> I would rather have a filter in my email client for them than > >> >>>> disabling > >> >>>> all of the alerting stuff > >> >>>> > >> >>>> /Johan > >> >>>> > >> >>>> On August 28, 2018 22:36:34 Douglas Duckworth > >> >>>> <dod2014(a)med.cornell.edu> > >> >>>> wrote: > >> >>>>> > >> >>>>> Hi > >> >>>>> > >> >>>>> Can someone please help? I keep getting ovirt alerts via email > >> >>>>> despite > >> >>>>> turning off postix and ovirt-engine-notifier.service > >> >>>>> > >> >>>>> Thanks, > >> >>>>> > >> >>>>> Douglas Duckworth, MSc, LFCS > >> >>>>> HPC System Administrator > >> >>>>> Scientific Computing Unit > >> >>>>> Weill Cornell Medicine > >> >>>>> 1300 York Avenue > >> >>>>> New York, NY 10065 > >> >>>>> E: doug(a)med.cornell.edu > >> >>>>> O: 212-746-6305 > >> >>>>> F: 212-746-8690 > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> On Fri, Aug 24, 2018 at 8:59 AM, Douglas Duckworth > >> >>>>> <dod2014(a)med.cornell.edu> wrote: > >> >>>>>> > >> >>>>>> Hi > >> >>>>>> > >> >>>>>> How do I turn off hosted engine alerts? We are in a testing > phase > >> >>>>>> so > >> >>>>>> these are not needed. I have disabled postfix on all hosts as > well > >> >>>>>> as > >> >>>>>> stopped the ovirt notification daemon on the hosted engine. I > kept > >> >>>>>> it > >> >>>>>> running while putting /dev/null in > >> >>>>>> > >> >>>>>> /usr/share/ovirt-engine/services/ovirt-engine- > notifier/ovirt-engine-notifier.conf > >> >>>>>> for mail server. Yet I still get alerts for every thing done > such > >> >>>>>> as > >> >>>>>> putting hosts in maintenance mode. Very confusing. > >> >>>>> > >> >>>>> > >> >>>>> _______________________________________________ > >> >>>>> Users mailing list -- users(a)ovirt.org > >> >>>>> To unsubscribe send an email to users-leave(a)ovirt.org > >> >>>>> Privacy Statement: > >> >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ovirt.org_site_privacy-2Dpolicy_&d=DwIFaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= > KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s=HNCnpn- > PS346coC2krxJr3kEOccwU9mSnRZNLj3IRok&e= > >> >>>>> oVirt Code of Conduct: > >> >>>>> > >> >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ovirt.org_community_about_community-2Dguidelines_&d=DwIFaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s= > bLInNmwMa9F4yME1S7vL8Qntn5MypII4kTgMKkZP-5c&e= > >> >>>>> List Archives: > >> >>>>> > >> >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. > ovirt.org_archives_list_users-40ovirt.org_message_ > X6YRFWUAKYFY2HQF56HGUK3BPXJL2HBH_&d=DwIFaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= > KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s=WW0BX4m6z_ > FG0g68LRhSqZ0w6Chf6mqNuSrYpX0ICds&e= > >> >>>> > >> >>>> > >> >>> > >> >>> _______________________________________________ > >> >>> Users mailing list -- users(a)ovirt.org > >> >>> To unsubscribe send an email to users-leave(a)ovirt.org > >> >>> Privacy Statement: > >> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ovirt.org_site_privacy-2Dpolicy_&d=DwIFaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= > KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s=HNCnpn- > PS346coC2krxJr3kEOccwU9mSnRZNLj3IRok&e= > >> >>> oVirt Code of Conduct: > >> >>> > >> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ovirt.org_community_about_community-2Dguidelines_&d=DwIFaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s= > bLInNmwMa9F4yME1S7vL8Qntn5MypII4kTgMKkZP-5c&e= > >> >>> List Archives: > >> >>> > >> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. > ovirt.org_archives_list_users-40ovirt.org_message_ > LIBA3OSRJUMGAGYOQDQAGLCYOWWNWJQ2_&d=DwIFaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= > KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s= > jfhpC57qilLXkNkXrEadoQk9KlDkD4UI7UjTzbyFUlg&e= > >> > > >> > > >> > > >> > _______________________________________________ > >> > Users mailing list -- users(a)ovirt.org > >> > To unsubscribe send an email to users-leave(a)ovirt.org > >> > Privacy Statement: > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ovirt.org_site_privacy-2Dpolicy_&d=DwIFaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= > KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s=HNCnpn- > PS346coC2krxJr3kEOccwU9mSnRZNLj3IRok&e= > >> > oVirt Code of Conduct: > >> > > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > ovirt.org_community_about_community-2Dguidelines_&d=DwIFaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s= > bLInNmwMa9F4yME1S7vL8Qntn5MypII4kTgMKkZP-5c&e= > >> > List Archives: > >> > > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. > ovirt.org_archives_list_users-40ovirt.org_message_ > I7EXHCE43XXSVKFVF5OCZACUCK4WW4UG_&d=DwIFaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m= > KNBFLrOsq9Nau2RNAaX-U7ES91Cg8z5gQzVkdVotcCk&s=AD1Knu9_62ziVk_ > XzUnjoQCT9CIbwnAE27GscnXM0M0&e= > >> > >

1 0

ovirt-engine-extension-aaa-ldap-setup credentials invalid
by Douglas Duckworth 30 Aug '18

30 Aug '18

Hi I am not able to finish ovirt-engine-extension-aaa-ldap-setup due to the error "credentials invalid." My password's correct. Our directory server's OpenLDAP standard schema. Our accounts are under OU=People. We are binding with a service account that has access to the entire DN. Any way to discern more information from attached logs? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690

2 1

Multiple CephFS Monitors cause issues with oVirt
by Stack Korora 30 Aug '18

30 Aug '18

Greetings, My setup is a complete Red Hat install. Manager OS: RHEL 7.5 Hypervisors OS: RHEL 7.5 Running Red Hat CephFS (with their Ceph repos on all of the systems) with Red Hat Virtualization (aka oVirt). Everything is fully patched and updated as of yesterday morning. Yes, I have valid Red Hat support but I figured this was an odd enough problem that the community (and the Red-Hat-ers who hang out on this list) might have a better idea of where to start. (Although I might open a ticket anyway just because that is what support is for, right? :) Quick background: Your /etc/fstab when you mount a nfs should probably look something like this: <your.nfs.ip.addr>:/path/ /mount/point nfs <various options> 0 0 Just one IP is needed. Since part of the redundancy for Ceph is in the monitors, to mount CephFS the fstab should look something like this: <your.ceph.ip.addr1>,<your.ceph.ip.addr2>,<your.ceph.ip.addrX>:/path/ /mount/point ceph <various options> 0 0 Both the Ceph community and Red Hat recommend the comma separator for mounting multiple CephFS monitor nodes. (See section 4.2 point 3) https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/c… Now to oVirt/RHV. When I mount my Data Domain path as a Posix file system with a path of "<your.ceph.ip.addr1>:/path/" it works splendidly well (especially after the last Red Hat kernel update!). I've done a bunch of stuff to it and it seems to work every time. However, I don't have the redundancy of multiple Ceph Monitors. When I mount my Data Domain path as a Posix file system with a path of "<your.ceph.ip.addr1>,<your.ceph.ip.addr2>,<your.ceph.ip.addrX>:/path/" most things seem to work. But I noticed a higher rate of failures. The only failure that I can trigger 100% of the time though is to mount a second data import domain and attempt to copy a vm disk from the import into the CephFS Data domain. Then I get an error like this: would VDSM ovirt01 command HSMGetAllTasksStatusesVDS failed: low level Image copy failed: (u'Destination volume 7c1bb510-9f35-4456-8d51-0955f788ac3e error: ParamsList: sep , in /rhev/data-center/mnt/<your.ceph.ip.addr1>,<your.ceph.ip.addr2>,<your.ceph.ip.addr3>:_ovirt_data/70fb34ad-e66d-43e6-8412-5e020baa34df/images/23991a68-0c43-433f-b1f9-48b1533da54a',) Uh, oh. It seems that the commas in the mount path are causing the problems. So I went looking through the logs for "sep , in" and found a bunch more hits which makes me think that this is actually the problem message. I've switched back to just one IP address for the time being but I obviously want the Ceph redundancy back. While running on just one IP, the vm disk that refused to copy before had no problem copying. The _only_ change I made was dropping two of the three IP's from the Data Domain path option. Is this a bug, or did I do something wrong? Does anyone have a suggestion for me to try? Thank you! ~Stack~

4 8

Re: Turn Off Email Alerts
by Douglas Duckworth 30 Aug '18

30 Aug '18

Yes, indeed! How can I change this internal Python setting? On Wed, Aug 29, 2018, 5:43 PM Martin Sivak <msivak(a)redhat.com> wrote: > Hi, > > two clarifications: > > Hosted engine is sending those emails using built-in Python SMTP > client that talks directly yo the SMTP server specified during install > time. We default to localhost, but you might have changed it. > > > notify.state_transition : maintenance|start|stop|migrate|up|down, type : > broker > > The value here is a regular expression that is matched against the > state transition string in the email. > > Best regards > > Martin Sivak > > On Wed, Aug 29, 2018 at 10:34 PM, Douglas Duckworth > <dod2014(a)med.cornell.edu> wrote: > > Thanks for sharing > > > > I may want to do that > > > > Though first I want to understand how the emails are arriving. > > > > I stooped ovirt-engine-notifier.service and postfix.service on all hosts > and > > the hosted engine. So how are the email being delivered? They are not > > running senmail so I don't understand what daemon's sending these > messages. > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York Avenue > > New York, NY 10065 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > > > On Wed, Aug 29, 2018 at 3:16 PM, Simone Tiraboschi <stirabos(a)redhat.com> > > wrote: > >> > >> Hi, > >> you can change the list of status you want to be notified about with > >> hosted-engine --set-shared-config notify.state_transition > >> The default is: > >> > >> [root@hehost01 ~]# hosted-engine --get-shared-config > >> notify.state_transition --type=broker > >> > >> notify.state_transition : maintenance|start|stop|migrate|up|down, type : > >> broker > >> > >> > >> On Wed, Aug 29, 2018 at 7:47 PM Douglas Duckworth > >> <dod2014(a)med.cornell.edu> wrote: > >>> > >>> I agree however we are in testing phase so I make changes a lot. > >>> Therefore alerts are not presently needed. > >>> > >>> So how to I turn them off? > >>> > >>> These steps do not work on hosted engine: > >>> > >>> > >>> me@ovirt-engine[~]$ sudo systemctl stop postfix.service > >>> me@ovirt-engine[~]$ sudo systemctl stop ovirt-engine-notifier.service > >>> me@ovirt-engine[~]$ sudo systemctl status > ovirt-engine-notifier.service > >>> ● ovirt-engine-notifier.service - oVirt Engine Notifier > >>> Loaded: loaded > (/usr/lib/systemd/system/ovirt-engine-notifier.service; > >>> enabled; vendor preset: disabled) > >>> Active: inactive (dead) since Wed 2018-08-29 13:41:55 EDT; 3s ago > >>> Process: 1814 > >>> > ExecStart=/usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.py > >>> --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, > >>> status=0/SUCCESS) > >>> Main PID: 1814 (code=exited, status=0/SUCCESS) > >>> > >>> Aug 25 12:09:31 ovirt-engine.pbtech systemd[1]: Starting oVirt Engine > >>> Notifier... > >>> Aug 25 12:09:33 ovirt-engine.pbtech systemd[1]: Started oVirt Engine > >>> Notifier. > >>> Aug 29 13:41:54 ovirt-engine.pbtech systemd[1]: Stopping oVirt Engine > >>> Notifier... > >>> Aug 29 13:41:55 ovirt-engine.pbtech systemd[1]: Stopped oVirt Engine > >>> Notifier. > >>> > >>> I still get an email every time I put a host in maint mode for example. > >>> > >>> > >>> Thanks, > >>> > >>> Douglas Duckworth, MSc, LFCS > >>> HPC System Administrator > >>> Scientific Computing Unit > >>> Weill Cornell Medicine > >>> 1300 York Avenue > >>> New York, NY 10065 > >>> E: doug(a)med.cornell.edu > >>> O: 212-746-6305 > >>> F: 212-746-8690 > >>> > >>> > >>> > >>> On Tue, Aug 28, 2018 at 4:43 PM, Johan Bernhardsson <johan(a)kafit.se> > >>> wrote: > >>>> > >>>> Those alerts are also coming from hosted-engine that keeps ovirt > manager > >>>> running. > >>>> > >>>> I would rather have a filter in my email client for them than > disabling > >>>> all of the alerting stuff > >>>> > >>>> /Johan > >>>> > >>>> On August 28, 2018 22:36:34 Douglas Duckworth < > dod2014(a)med.cornell.edu> > >>>> wrote: > >>>>> > >>>>> Hi > >>>>> > >>>>> Can someone please help? I keep getting ovirt alerts via email > despite > >>>>> turning off postix and ovirt-engine-notifier.service > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Douglas Duckworth, MSc, LFCS > >>>>> HPC System Administrator > >>>>> Scientific Computing Unit > >>>>> Weill Cornell Medicine > >>>>> 1300 York Avenue > >>>>> New York, NY 10065 > >>>>> E: doug(a)med.cornell.edu > >>>>> O: 212-746-6305 > >>>>> F: 212-746-8690 > >>>>> > >>>>> > >>>>> > >>>>> On Fri, Aug 24, 2018 at 8:59 AM, Douglas Duckworth > >>>>> <dod2014(a)med.cornell.edu> wrote: > >>>>>> > >>>>>> Hi > >>>>>> > >>>>>> How do I turn off hosted engine alerts? We are in a testing phase > so > >>>>>> these are not needed. I have disabled postfix on all hosts as well > as > >>>>>> stopped the ovirt notification daemon on the hosted engine. I kept > it > >>>>>> running while putting /dev/null in > >>>>>> > /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf > >>>>>> for mail server. Yet I still get alerts for every thing done such > as > >>>>>> putting hosts in maintenance mode. Very confusing. > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Users mailing list -- users(a)ovirt.org > >>>>> To unsubscribe send an email to users-leave(a)ovirt.org > >>>>> Privacy Statement: > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri… > >>>>> oVirt Code of Conduct: > >>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit… > >>>>> List Archives: > >>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv… > >>>> > >>>> > >>> > >>> _______________________________________________ > >>> Users mailing list -- users(a)ovirt.org > >>> To unsubscribe send an email to users-leave(a)ovirt.org > >>> Privacy Statement: > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri… > >>> oVirt Code of Conduct: > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit… > >>> List Archives: > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv… > > > > > > > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri… > > oVirt Code of Conduct: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit… > > List Archives: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv… > > >

2 1

Re: Turn Off Email Alerts
by Douglas Duckworth 29 Aug '18

29 Aug '18

Thanks for sharing I may want to do that Though first I want to understand how the emails are arriving. I stooped ovirt-engine-notifier.service and postfix.service on all hosts and the hosted engine. So how are the email being delivered? They are not running senmail so I don't understand what daemon's sending these messages. Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Wed, Aug 29, 2018 at 3:16 PM, Simone Tiraboschi <stirabos(a)redhat.com> wrote: > Hi, > you can change the list of status you want to be notified about > with hosted-engine --set-shared-config notify.state_transition > The default is: > > [root@hehost01 ~]# hosted-engine --get-shared-config > notify.state_transition --type=broker > > notify.state_transition : maintenance|start|stop|migrate|up|down, type : > broker > > > On Wed, Aug 29, 2018 at 7:47 PM Douglas Duckworth <dod2014(a)med.cornell.edu> > wrote: > >> I agree however we are in testing phase so I make changes a lot. >> Therefore alerts are not presently needed. >> >> So how to I turn them off? >> >> These steps do not work on hosted engine: >> >> >> me@ovirt-engine[~]$ sudo systemctl stop postfix.service >> me@ovirt-engine[~]$ sudo systemctl stop ovirt-engine-notifier.service >> me@ovirt-engine[~]$ sudo systemctl status ovirt-engine-notifier.service >> ● ovirt-engine-notifier.service - oVirt Engine Notifier >> Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-notifier.service; >> enabled; vendor preset: disabled) >> Active: inactive (dead) since Wed 2018-08-29 13:41:55 EDT; 3s ago >> Process: 1814 ExecStart=/usr/share/ovirt-engine/services/ovirt-engine- >> notifier/ovirt-engine-notifier.py --redirect-output --systemd=notify >> $EXTRA_ARGS start (code=exited, status=0/SUCCESS) >> Main PID: 1814 (code=exited, status=0/SUCCESS) >> >> Aug 25 12:09:31 ovirt-engine.pbtech systemd[1]: Starting oVirt Engine >> Notifier... >> Aug 25 12:09:33 ovirt-engine.pbtech systemd[1]: Started oVirt Engine >> Notifier. >> Aug 29 13:41:54 ovirt-engine.pbtech systemd[1]: Stopping oVirt Engine >> Notifier... >> Aug 29 13:41:55 ovirt-engine.pbtech systemd[1]: Stopped oVirt Engine >> Notifier. >> >> I still get an email every time I put a host in maint mode for example. >> >> >> Thanks, >> >> Douglas Duckworth, MSc, LFCS >> HPC System Administrator >> Scientific Computing Unit >> Weill Cornell Medicine >> 1300 York Avenue >> <https://maps.google.com/?q=1300+York+Avenue+%0D%0A+New+York,+NY+10065&entry…> >> New York, NY 10065 >> <https://maps.google.com/?q=1300+York+Avenue+%0D%0A+New+York,+NY+10065&entry…> >> E: doug(a)med.cornell.edu >> O: 212-746-6305 >> F: 212-746-8690 >> >> >> >> On Tue, Aug 28, 2018 at 4:43 PM, Johan Bernhardsson <johan(a)kafit.se> >> wrote: >> >>> Those alerts are also coming from hosted-engine that keeps ovirt manager >>> running. >>> >>> I would rather have a filter in my email client for them than disabling >>> all of the alerting stuff >>> >>> /Johan >>> >>> On August 28, 2018 22:36:34 Douglas Duckworth <dod2014(a)med.cornell.edu> >>> wrote: >>> >>>> Hi >>>> >>>> Can someone please help? I keep getting ovirt alerts via email despite >>>> turning off postix and ovirt-engine-notifier.service >>>> >>>> Thanks, >>>> >>>> Douglas Duckworth, MSc, LFCS >>>> HPC System Administrator >>>> Scientific Computing Unit >>>> Weill Cornell Medicine >>>> 1300 York Avenue >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3…> >>>> New York, NY 10065 >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3…> >>>> E: doug(a)med.cornell.edu >>>> O: 212-746-6305 >>>> F: 212-746-8690 >>>> >>>> >>>> >>>> On Fri, Aug 24, 2018 at 8:59 AM, Douglas Duckworth < >>>> dod2014(a)med.cornell.edu> wrote: >>>> >>>>> Hi >>>>> >>>>> How do I turn off hosted engine alerts? We are in a testing phase so >>>>> these are not needed. I have disabled postfix on all hosts as well as >>>>> stopped the ovirt notification daemon on the hosted engine. I kept it >>>>> running while putting /dev/null in /usr/share/ovirt-engine/ >>>>> services/ovirt-engine-notifier/ovirt-engine-notifier.conf for mail >>>>> server. Yet I still get alerts for every thing done such as putting hosts >>>>> in maintenance mode. Very confusing. >>>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >>>> guidelines/ >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ >>>> message/X6YRFWUAKYFY2HQF56HGUK3BPXJL2HBH/ >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >>>> >>> >>> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >> guidelines/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ >> message/LIBA3OSRJUMGAGYOQDQAGLCYOWWNWJQ2/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >> >

2 1

Re: Turn Off Email Alerts
by Douglas Duckworth 29 Aug '18

29 Aug '18

I agree however we are in testing phase so I make changes a lot. Therefore alerts are not presently needed. So how to I turn them off? These steps do not work on hosted engine: me@ovirt-engine[~]$ sudo systemctl stop postfix.service me@ovirt-engine[~]$ sudo systemctl stop ovirt-engine-notifier.service me@ovirt-engine[~]$ sudo systemctl status ovirt-engine-notifier.service ● ovirt-engine-notifier.service - oVirt Engine Notifier Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-notifier.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2018-08-29 13:41:55 EDT; 3s ago Process: 1814 ExecStart=/usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.py --redirect-output --systemd=notify $EXTRA_ARGS start (code=exited, status=0/SUCCESS) Main PID: 1814 (code=exited, status=0/SUCCESS) Aug 25 12:09:31 ovirt-engine.pbtech systemd[1]: Starting oVirt Engine Notifier... Aug 25 12:09:33 ovirt-engine.pbtech systemd[1]: Started oVirt Engine Notifier. Aug 29 13:41:54 ovirt-engine.pbtech systemd[1]: Stopping oVirt Engine Notifier... Aug 29 13:41:55 ovirt-engine.pbtech systemd[1]: Stopped oVirt Engine Notifier. I still get an email every time I put a host in maint mode for example. Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Tue, Aug 28, 2018 at 4:43 PM, Johan Bernhardsson <johan(a)kafit.se> wrote: > Those alerts are also coming from hosted-engine that keeps ovirt manager > running. > > I would rather have a filter in my email client for them than disabling > all of the alerting stuff > > /Johan > > On August 28, 2018 22:36:34 Douglas Duckworth <dod2014(a)med.cornell.edu> > wrote: > >> Hi >> >> Can someone please help? I keep getting ovirt alerts via email despite >> turning off postix and ovirt-engine-notifier.service >> >> Thanks, >> >> Douglas Duckworth, MSc, LFCS >> HPC System Administrator >> Scientific Computing Unit >> Weill Cornell Medicine >> 1300 York Avenue >> <https://maps.google.com/?q=1300+York+Avenue+%0D%0A+New+York,+NY+10065&entry…> >> New York, NY 10065 >> <https://maps.google.com/?q=1300+York+Avenue+%0D%0A+New+York,+NY+10065&entry…> >> E: doug(a)med.cornell.edu >> O: 212-746-6305 >> F: 212-746-8690 >> >> >> >> On Fri, Aug 24, 2018 at 8:59 AM, Douglas Duckworth < >> dod2014(a)med.cornell.edu> wrote: >> >>> Hi >>> >>> How do I turn off hosted engine alerts? We are in a testing phase so >>> these are not needed. I have disabled postfix on all hosts as well as >>> stopped the ovirt notification daemon on the hosted engine. I kept it >>> running while putting /dev/null in /usr/share/ovirt-engine/ser >>> vices/ovirt-engine-notifier/ovirt-engine-notifier.conf for mail >>> server. Yet I still get alerts for every thing done such as putting hosts >>> in maintenance mode. Very confusing. >>> >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >> guidelines/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ >> message/X6YRFWUAKYFY2HQF56HGUK3BPXJL2HBH/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >> > >

2 1

Re: Weird Memory Leak Issue
by Robert O'Kane 29 Aug '18

29 Aug '18

Not for me. After restarting the engine,it doesn't matter if I restart the Hypervisors, I only get the "protocol=gluster" when I restart the VMs... migration is not enough. "virsh -r dumpxml <VM-ID>" Cheers, Robert O'Kane On 08/29/2018 06:54 PM, Edward Clay wrote: > Not sure how you're setup. I only had to migrate the VMs off to another hypervisor then put the hypervisor in maintenance which seems to unmount the one gluster volume. After upgrading, rebooting and activating the HV I could migrate VMs back to it. This seemed to work for me. > > ________________________________ > From: Robert O'Kane <okane(a)khm.de> > Sent: Wednesday, August 29, 2018 10:42:56 AM > To: users(a)ovirt.org > Subject: [ovirt-users] Re: Weird Memory Leak Issue > > **Security Notice - This external email is NOT from The Hut Group** > > Ah, the FUSE mounts... I just saw last week that the upgrade to 4.2 removed the "LibgfApiSupported" flag by default. > > That is possibly why the leak simply appeared.... > > OK, I was wondering where this came from. Tomorrow I will upgrade and test. I still will have to eventually reboot the VMs to get the gluster mounts again.... :-/ > > > > On 08/29/2018 05:18 PM, Cole Johnson wrote: >> Great! I'll look for the update. >> >> On Wed, Aug 29, 2018 at 7:50 AM Darrell Budic <budic(a)onholyground.com> wrote: >>> >>> There’s a memory leak in gluster 3.12.9 - 3.12.12 on fuse mounted volumes, sounds like what you’re seeing. >>> >>> The fix is in 3.12.13, which should be showing up today or tomorrow in the centos repos (currently available from the testing repo). I’ve been running it overnight on one host to test, looks like they got it. >>> >>> ________________________________ >>> From: Cole Johnson <sizzlinsaguaro(a)gmail.com> >>> Subject: [ovirt-users] Weird Memory Leak Issue >>> Date: August 29, 2018 at 9:35:39 AM CDT >>> To: users(a)ovirt.org >>> >>> Hello, >>> I have a hyperconverged, self hosted ovirt cluster with three hosts, >>> running 4 VM's. The hosts are running the latest ovirt node. The >>> VM's are Linux, Windows server 2016, and Windows Server 2008r2. The >>> problem is with any host running the 2008r2 VM will run out of memory >>> after 8-10 hours, causing any VM on the host to be paused, and making >>> to host all but unresponsive. This problem seems to only exist with >>> this specific VM. None of the other running VM's have this problem. >>> I can resolve the problem by migrating the VM to a different host, >>> then putting the host into maintenance mode, the activating it back. >>> The leak appears to be in glusterfsd. Is there anything I can do to >>> permanently fix this? > > ______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/<https://www.ovirt.org/site/privacy-policy/> >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://www.ovirt.org/community/about/community-guidelines/> >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CLNILVLZ4D3BJ…<https://lists.ovirt.org/archives/list/users@ovirt.org/message/CLNILVLZ4D3BJ…> >> > > -- > Robert O'Kane > Systems Administrator > Kunsthochschule für Medien Köln > Peter-Welter-Platz 2 > 50676 Köln > > fon: +49(221)20189-223 > fax: +49(221)20189-49223 > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/<https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://www.ovirt.org/community/about/community-guidelines/> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/F7EY4FX7UOBGW…<https://lists.ovirt.org/archives/list/users@ovirt.org/message/F7EY4FX7UOBGW…> > > Edward Clay > Systems Administrator > The Hut Group<http://www.thehutgroup.com/> > > Tel: > Email: edward.clay(a)uk2group.com<mailto:edward.clay@uk2group.com> > > For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Fifth Floor, Voyager House, Chicago Avenue, Manchester Airport, M90 3DQ and/or any of its respective subsidiaries. > > Confidentiality Notice > This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 811888 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company. > > Encryptions and Viruses > Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail. > > Monitoring > Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes. > > hgvyjuv > -- Robert O'Kane Systems Administrator Kunsthochschule für Medien Köln Peter-Welter-Platz 2 50676 Köln fon: +49(221)20189-223 fax: +49(221)20189-49223

1 0

Weird Memory Leak Issue
by Cole Johnson 29 Aug '18

29 Aug '18

Hello, I have a hyperconverged, self hosted ovirt cluster with three hosts, running 4 VM's. The hosts are running the latest ovirt node. The VM's are Linux, Windows server 2016, and Windows Server 2008r2. The problem is with any host running the 2008r2 VM will run out of memory after 8-10 hours, causing any VM on the host to be paused, and making to host all but unresponsive. This problem seems to only exist with this specific VM. None of the other running VM's have this problem. I can resolve the problem by migrating the VM to a different host, then putting the host into maintenance mode, the activating it back. The leak appears to be in glusterfsd. Is there anything I can do to permanently fix this?

4 4

Next Gluster Updates?
by Robert O'Kane 29 Aug '18

29 Aug '18

I had a bug request in Bugzilla for Gluster being killed due to a memory leak. The Gluster People say it is fixed in gluster-3.12.13 When will Ovirt have this update? I am getting tired of having to restart my hypervisors every week or so... I currently have ovirt-release42-4.2.5.1-1.el7.noarch and yum check-updates shows me no new gluster versions.....(still 3.12.11) Cheers, Robert O'Kane -- Robert O'Kane Systems Administrator Kunsthochschule für Medien Köln Peter-Welter-Platz 2 50676 Köln fon: +49(221)20189-223 fax: +49(221)20189-49223

3 2

Windows 10 vs others windows
by carl langlois 29 Aug '18

29 Aug '18

Hi Why when en try to do a Windows 10 machine i have a error on the cpu guest os not supported but not windows 8 or 7? Thanks,

2 1

snapshots upload
by David David 29 Aug '18

29 Aug '18

hi all ovirt engine 4.2.5.2-1.el7 ovirt node: KVM Version: 2.9.0 - 16.el7_4.14.1 LIBVIRT Version: libvirt-3.2.0-14.el7_4.9 VDSM Version: vdsm-4.20.27.1-1.el7.centos Can't restore vm by following this instruction https://ovirt.org/develop/release-management/features/storage/backup-restor… error message: # python upload_disk_snapshots.py Creating disk: 414d6613-5cfe-493c-ae6c-aa29caa32983 Traceback (most recent call last): File "upload_disk_snapshots.py", line 305, in <module> disk = create_disk(base_volume, disk_id, sd_name, disks_service) File "upload_disk_snapshots.py", line 186, in create_disk name=sd_name File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 6715, in add return self._internal_add(disk, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Virtual Disk. Disk configuration (RAW Sparse) is incompatible with the storage domain type.]". HTTP response code is 400. # tree 414d6613-5cfe-493c-ae6c-aa29caa32983/ 414d6613-5cfe-493c-ae6c-aa29caa32983/ ├── 3610d5fd-6f55-46d9-a226-c06eee8e21e6 └── f77207b2-6e5b-4464-bd6f-5ae6d776435d 414d6613-5cfe-493c-ae6c-aa29caa32983 - disk id 3610d5fd-6f55-46d9-a226-c06eee8e21e6 - base image file f77207b2-6e5b-4464-bd6f-5ae6d776435d - snapshot1 file # qemu-img info 414d6613-5cfe-493c-ae6c-aa29caa32983/3610d5fd-6f55-46d9-a226-c06eee8e21e6 image: 414d6613-5cfe-493c-ae6c-aa29caa32983/3610d5fd-6f55-46d9-a226-c06eee8e21e6 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 22G cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # qemu-img info 414d6613-5cfe-493c-ae6c-aa29caa32983/f77207b2-6e5b-4464-bd6f-5ae6d776435d image: 414d6613-5cfe-493c-ae6c-aa29caa32983/f77207b2-6e5b-4464-bd6f-5ae6d776435d file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 1.0G cluster_size: 65536 backing file: 3610d5fd-6f55-46d9-a226-c06eee8e21e6 (actual path: 414d6613-5cfe-493c-ae6c-aa29caa32983/3610d5fd-6f55-46d9-a226-c06eee8e21e6) backing file format: qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false upload_disk_snapshots.py: ========================== if __name__ == "__main__": # Set storage domain name sd_name = 'data_sas3' # Set OVF file path ovf_file_path = 'f4fdaf18-b944-4d22-879b-e235145a93f6.ovf' # Disk to upload disk_path = '414d6613-5cfe-493c-ae6c-aa29caa32983' disk_id = os.path.basename(disk_path) ==========================

2 1

Guide for setting up GlusterFS geo-replication?
by Jayme 29 Aug '18

29 Aug '18

Is there an updated guide for setting up GlusterFS geo-replication? What I am interested in is having another oVirt setup on a separate server with glusterFS volume replicated to it. If my primary cluster went down I would be able to start important VMs on the secondary oVirt build until I'm able to repair the main cluster. In oVirt 4.2.5 clicking geo-replication > new in the volume GUI produces no results. Nothing happens when I click it (using multiple browsers). And under storage domains clicking remote data-sync -> setup produces a popup but just has a loading icon in the middle and never completes loading. Why are neither of these two features working for me?

4 8

Turn Off Email Alerts
by Douglas Duckworth 28 Aug '18

28 Aug '18

Hi How do I turn off hosted engine alerts? We are in a testing phase so these are not needed. I have disabled postfix on all hosts as well as stopped the ovirt notification daemon on the hosted engine. I kept it running while putting /dev/null in /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf for mail server. Yet I still get alerts for every thing done such as putting hosts in maintenance mode. Very confusing.

2 2

Re: SSSD on Hosted Engine
by Douglas Duckworth 28 Aug '18

28 Aug '18

It says I am not authorized. Can you give me permission to comment? My bugzilla account would be doug(a)med.cornell.edu. Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Wed, Aug 22, 2018 at 10:44 AM, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote: > > > 2018-08-22 14:59 GMT+02:00 Douglas Duckworth <dod2014(a)med.cornell.edu>: > >> Yay, I was able to restore nsswitch so things now work. >> >> Anyway that oVirt can use SSSD for web auth? >> >> Per https://ovirt.org/develop/release-management/features/infra/aaa_faq/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ovirt.org_develop_rele…> >> there's a bug https://bugzilla.redhat.com/show_bug.cgi?id=829292 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.redhat.com_sh…> >> that prevents it from working? >> > > I would suggest to comment on that bug with your use case, it will help > understanding the needs. > > > > > > >> >> >> >> Thanks, >> >> Douglas Duckworth, MSc, LFCS >> HPC System Administrator >> Scientific Computing Unit >> Weill Cornell Medicine >> 1300 York - LC-502 >> E: doug(a)med.cornell.edu >> O: 212-746-6305 >> F: 212-746-8690 >> >> >> On Wed, Aug 22, 2018 at 8:51 AM, Douglas Duckworth < >> dod2014(a)med.cornell.edu> wrote: >> >>> Hi >>> >>> I am trying to configure sssd on my hosted engine. Essentially we >>> control host access in LDAP so I want sssd to read that thus allow my >>> coworkers to login to hosted engine vm. >>> >>> For some reason sssd reports backend offline even though it's >>> resolvable, pingable, with ports open. I see that it's a SELinux issue >>> which I can resolve. After changing to permissive SSSD works. >>> >>> To have system read sssd database I set hosts line in /etc/nsswitch.conf >>> to: >>> >>> hosts files sss >>> >>> Though it seems that I did something bad to /etc/nsswitch.conf as now >>> yum, ping, etc does not work. >>> >>> Could someone suggest how to restore this file or could anyone share >>> theirs? >>> >>> Thanks, >>> >>> Douglas Duckworth, MSc, LFCS >>> HPC System Administrator >>> Scientific Computing Unit >>> Weill Cornell Medicine >>> 1300 York - LC-502 >>> E: doug(a)med.cornell.edu >>> O: 212-746-6305 >>> F: 212-746-8690 >>> >>> >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >> oVirt Code of Conduct: https://www.ovirt.org/communit >> y/about/community-guidelines/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >> List Archives: https://lists.ovirt.org/archiv >> es/list/users(a)ovirt.org/message/DSFBEFKBNAWZMLMASRJ7YKZO3PZWKJLV/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >> >> > > > -- > > SANDRO BONAZZOLA > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_&d=DwMF…> > > sbonazzo(a)redhat.com > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__red.ht_sig&d=DwMFaQ&c=…> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.redhat.com_en_even…> >

2 1

mask or disabled lldpad service
by Klaas Demter 28 Aug '18

28 Aug '18

Hi, I have a QLogic Corp. QLogic 2x1GE+2x10GE QL41162HMRJ CNA network card. This network card comes with it's own lldp implementation inside their management firmware. Running two lldp agents seems to create several issues (http://lists.us.dell.com/pipermail/linux-poweredge/2018-July/051860.html) The QLogic engineers said I need to disable lldpad in the operating system. This leads me here, what is lldp actually used for within ovirt? If it is actually needed how could I deal with a card that does not support lldpad but rather has it's own implementation? Greetings Klaas

2 1

ovirt selfhost error
by mustafa.taha.mu95＠gmail.com 28 Aug '18

28 Aug '18

hi , i am a new user of ovirt , i want to setup self host , i follow the step in https://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hoste... , after i configure storage this error appear fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"} how can i fixed it ?

3 6

[ANN] oVirt 4.2.6 Third Release Candidate is now available
by Sandro Bonazzola 28 Aug '18

28 Aug '18

The oVirt Project is pleased to announce the availability of the oVirt 4.2.6 third Release Candidate, as of August 28th, 2018. This update is a release candidate of the sixth in a series of stabilization updates to the 4.2 series. This is pre-release software. This pre-release should not to be used in production. This release is available now for: * Red Hat Enterprise Linux 7.5 or later * CentOS Linux (or similar) 7.5 or later This release supports Hypervisor Hosts running: * Red Hat Enterprise Linux 7.5 or later * CentOS Linux (or similar) 7.5 or later See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is available - oVirt Node is available [2] - oVirt Windows Guest Tools is available [2] Additional Resources: * Read more about the oVirt 4.2.6 release highlights: http://www.ovirt.org/release/4.2.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.2.6/ [2] http://resources.ovirt.org/pub/ovirt-4.2-pre/iso/ -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f…>

1 0

VDO + HCI = failed reboots
by Donny Davis 28 Aug '18

28 Aug '18

I just spun up the latest and greatest ovirt has to offer, and I am building out an HCI cluster. The deployment went wonderfully. I had dns setup for everything, and it just worked. Great job team! I just wanted to add in something i noticed after rebooting the nodes. The gluster service, and volumes would not come back up after rebooting. I found that there needs to be an entry in the fstab for the gluster logical volumes to tell them to start the vdo service. So without this, the node will fail to boot if you are using vdo. inode64,noatime,nodiratime,discard,x-systemd.requires=vdo.service the defaults are this inode64,noatime,nodiratime Hopefully it helps someone else :) Cheers

2 4

hosted-engine --deploy erorr
by mustafa.taha.mu95＠gmail.com 28 Aug '18

28 Aug '18

hi , i am a new user of ovirt , i want to setup self host , i follow the step in https://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_… , after i configure storage this error appear fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"} how can i fixed it ?

1 0

Migrating to HCI
by Vincent Royer 28 Aug '18

28 Aug '18

I have 2 hosts using NFS storage I want to migrate to HCI and add a third host. Have added the 10Gbe cards and drives to the 3 target hosts and setup a separate physical network for storage. Can I: - deploy a single-node HCI to host 3 - export the VMs to an NFS export domain - Import the VMs to the HCI node - wipe the first 2 hosts and re-deploy them to the HCI DC Is there a guide for this? or should I just export the VMs, blow away all the nodes, and deploy as a 3-node HCI using the wizards? I'm hoping to minimize downtime of the VMs to under one day. *Vince*

2 1

One Windows VM is not responding
by Jonathan Baecker 27 Aug '18

27 Aug '18

Hello Everybody! We have here 13 VMs running under ovirt 4.2.4. Two of them are Windows Server 2016. On one runs a AD, DNS and one application. On the second one runs an SQL Server and also an application. This second one have the problem, that periodical it goes in a state where I get in ovirt the message: *not responding * Beside this the VM is running normal. I can connect it over Remote Desktop. But I can not connect it with ovirt/noVNC, the button is gray out. In a weekly cycle it runs a backup script, maybe this brings some problems, but I really don't know how to debug this. It can be that the VM runs for 3 weeks normal, and then it goes in this state, so I can not really say when this is happen. Also the fact that the other Windows Server VM runs normal, wonders me. Do you have experienced this problem, or do you know how I found out the issue? Best Regards! Jonathan

2 2

Re: External ceph storage
by Luca 'remix_tj' Lorenzetto 27 Aug '18

27 Aug '18

No, The only way you have is to configure cinder to manage ceph pool or in alternative you have to deploy an iscsi gateway, no other ways are available at the moment. So you can't use rbd directly. Luca Il dom 27 mag 2018, 16:54 Leo David <leoalex(a)gmail.com> ha scritto: > Thank you Luca, > At the moment i would try the cinder storage provider, since we already > have a proxmox cluster directly connecting to ceph. The problem is that I > just could not find a straight way to do this. > ie: Specify the ceph monitors and ceph pool to connect to. Can oVirt > directly connect to ceph monitors ? How the configuration should be done if > so ? > Thank you very much ! > > > On Sun, May 27, 2018, 17:20 Luca 'remix_tj' Lorenzetto < > lorenzetto.luca(a)gmail.com> wrote: > >> Hello, >> >> Yes, using cinder or through iscsi gateway. >> >> For a simpler setup i suggest the second option. >> >> Luca >> >> Il dom 27 mag 2018, 16:08 Leo David <leoalex(a)gmail.com> ha scritto: >> >>> Hello everyone, >>> I am new to ovirt and very impressed of its features. I would like to >>> levereage on our existing ceph cluster to provide rbd images for vm hdds, >>> is this possible to achieve ? >>> Thank you very much ! >>> Regards, >>> Leo >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> >>

3 3

Unknown CPU model Broadwell-IBRS-SSBD
by me＠brendanh.com 24 Aug '18

24 Aug '18

Hi, I've just installed hosted-engine. It proceeds to near the end of the Ansible script, then pauses at: "TASK [Check engine VM health]" then eventually errors: "Engine VM is not running, please check vdsm logs. The VDSM logs have error: "libvirtError: internal error: Unknown CPU model Broadwell-IBRS-SSBD" Also, in the oVirt Node webui, the HostedEngine VM is created, but not started. If I press Run, the following error is shown: VM START action failed error: Failed to start domain HostedEngine error: internal error: Unknown CPU model Broadwell-IBRS-SSBD I notice a Trello card was created to add similar Skylake & EPYC CPUs, but not Broadwell: https://trello.com/c/5nGTgQ9P/89-new-ibrs-and-ssbd-cpu-types# Does the Broadwell-IBRS-SSBD CPU need to be added to this supported types list? Many thanks.

3 4

Re: losing ib0 connection after activating host
by Douglas Duckworth 24 Aug '18

24 Aug '18

Sorry, I mean "migration network" for moving live migration traffic. FDR infiniband much faster than 1Gb network which currently acts as migration network, vm network, display network, mgmt network, etc. Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Fri, Aug 24, 2018 at 9:36 AM, Dominik Holler <dholler(a)redhat.com> wrote: > On Thu, 23 Aug 2018 13:51:39 -0400 > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > THANKS! > > > > ib0 now up with NFS storage back on this hypervisor > > > > Thanks for letting us know. > > > Though how do I make it a transfer network? I don't see an option. > > > > I do not understand the meaning of "transfer network". > The network interface to use for NFS results from the routing tables of > the host. > In "Compute > Clusters > Clustername > Logical Networks > Manage > Networks" network roles for some kind of loads can be assigned, but not > for NFS access. > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York - LC-502 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > On Thu, Aug 23, 2018 at 11:12 AM, Douglas Duckworth > > <dod2014(a)med.cornell.edu > > > wrote: > > > > > Hi Dominik > > > > > > Yes, the network-script was created by our Ansible role that deploys > > > CentOS hosts. It pulls the IP from DNS then templates the script > > > and copies to host. > > > > > > I will try this oVirt step then see if it works! > > > > > > Thanks, > > > > > > Douglas Duckworth, MSc, LFCS > > > HPC System Administrator > > > Scientific Computing Unit > > > Weill Cornell Medicine > > > 1300 York - LC-502 > > > E: doug(a)med.cornell.edu > > > O: 212-746-6305 > > > F: 212-746-8690 > > > > > > > > > On Thu, Aug 23, 2018 at 11:09 AM, Dominik Holler > > > <dholler(a)redhat.com> wrote: > > > > > >> Is ifcfg-ib0 created before adding the host? > > >> Can ib0 be reconfigured using engine, e.g. by > > >> "Compute > Hosts > hostx > Network Interfaces > Setup Host > > >> Networks"? If this some kind of self-hosted engine? > > >> > > >> On Thu, 23 Aug 2018 09:30:59 -0400 > > >> Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > >> > > >> > Here's a link to the files: > > >> > > > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_ > > >> 2wjZ6Vo&d=DwICAg&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu > > >> 2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=Y25- > > >> OOvgu58jlC82-fzBeNIpQ7ZscoHznffUhqE6EBM&s=QQXlC9Tisa60TvimyS > > >> 3BnFDCaDF7VPD8eCzT-Fke-p0&e= > > >> > > > >> > Thank you! > > >> > > > >> > Thanks, > > >> > > > >> > Douglas Duckworth, MSc, LFCS > > >> > HPC System Administrator > > >> > Scientific Computing Unit > > >> > Weill Cornell Medicine > > >> > 1300 York - LC-502 > > >> > E: doug(a)med.cornell.edu > > >> > O: 212-746-6305 > > >> > F: 212-746-8690 > > >> > > > >> > > > >> > On Thu, Aug 23, 2018 at 6:51 AM, Dominik Holler > > >> > <dholler(a)redhat.com> wrote: > > >> > > > >> > > Would you please share the vdsm.log and the supervdsm.log from > > >> > > this host? > > >> > > > > >> > > On Wed, 22 Aug 2018 11:36:09 -0400 > > >> > > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > >> > > > > >> > > > Hi > > >> > > > > > >> > > > I keep losing ib0 connection on hypervisor after adding host > > >> > > > to engine. This makes the host not really work since NFS > > >> > > > will be mounted over ib0. > > >> > > > > > >> > > > I don't really understand why this occurs. > > >> > > > > > >> > > > OS: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# cat /etc/redhat-release > > >> > > > CentOS Linux release 7.5.1804 (Core) > > >> > > > > > >> > > > Here's the network script: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# > > >> > > > cat /etc/sysconfig/network-scripts/ifcfg-ib0 DEVICE=ib0 > > >> > > > BOOTPROTO=static > > >> > > > IPADDR=172.16.0.207 > > >> > > > NETMASK=255.255.255.0 > > >> > > > ONBOOT=yes > > >> > > > ZONE=public > > >> > > > > > >> > > > When I try "ifup" > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# ifup ib0 > > >> > > > Error: Connection activation failed: No suitable device > > >> > > > found for this connection. > > >> > > > > > >> > > > The error in syslog: > > >> > > > > > >> > > > Aug 22 11:31:50 ovirt-hv2 kernel: IPv4: martian source > > >> > > > 172.16.0.87 from 172.16.0.49, on dev ib0 > > >> > > > Aug 22 11:31:53 ovirt-hv2 NetworkManager[1070]: <info> > > >> > > > [1534951913.7486] audit: op="connection-activate" > > >> > > > uuid="2ab4abde-b8a5-6cbc-19b1-2bfb193e4e89" name="System ib0" > > >> > > > result="fail" reason="No suitable device found for this > > >> > > > connection. > > >> > > > > > >> > > > As you can see media state up: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# ip a > > >> > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > >> > > > UNKNOWN group default qlen 1000 > > >> > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > >> > > > inet 127.0.0.1/8 scope host lo > > >> > > > valid_lft forever preferred_lft forever > > >> > > > 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq > > >> > > > master ovirtmgmt state UP group default qlen 1000 > > >> > > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > >> > > > 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > >> > > > state DOWN group default qlen 1000 > > >> > > > link/ether 50:9a:4c:89:d3:82 brd ff:ff:ff:ff:ff:ff > > >> > > > 4: p1p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc > > >> > > > mq state DOWN group default qlen 1000 > > >> > > > link/ether b4:96:91:13:ea:68 brd ff:ff:ff:ff:ff:ff > > >> > > > 5: p1p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc > > >> > > > mq state DOWN group default qlen 1000 > > >> > > > link/ether b4:96:91:13:ea:6a brd ff:ff:ff:ff:ff:ff > > >> > > > 6: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > >> > > > pfifo_fast state UNKNOWN group default qlen 1000 > > >> > > > link/ether 50:9a:4c:89:d3:84 brd ff:ff:ff:ff:ff:ff > > >> > > > inet 169.254.0.2/16 brd 169.254.255.255 scope global > > >> > > > idrac valid_lft forever preferred_lft forever > > >> > > > 7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq > > >> > > > state UP group default qlen 256 > > >> > > > link/infiniband > > >> > > > a0:00:02:08:fe:80:00:00:00:00:00:00:ec:0d:9a:03:00:1d:13:41 > > >> > > > brd > > >> > > > 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > > >> > > > 8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop > > >> > > > state DOWN group default qlen 1000 link/ether > > >> > > > 12:b4:30:22:39:5b brd ff:ff:ff:ff:ff:ff 9: br-int: > > >> > > > <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group > > >> > > > default qlen 1000 link/ether 3e:32:e6:66:98:49 brd > > >> > > > ff:ff:ff:ff:ff:ff 25: ovirtmgmt: > > >> > > > <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue > > >> > > > state UP group default qlen 1000 link/ether > > >> > > > 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff inet 10.0.0.183/16 > > >> > > > brd 10.0.255.255 scope global ovirtmgmt valid_lft forever > > >> > > > preferred_lft forever 26: genev_sys_6081: > > >> > > > <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue > > >> > > > master ovs-system state UNKNOWN group default qlen 1000 > > >> > > > link/ether aa:32:82:1b:01:d9 brd ff:ff:ff:ff:ff:ff > > >> > > > 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop > > >> > > > state DOWN group default qlen 1000 link/ether > > >> > > > 32:ff:5d:b8:c2:b4 brd ff:ff:ff:ff:ff:ff > > >> > > > > > >> > > > The card is FDR: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# lspci -v | grep Mellanox > > >> > > > 01:00.0 Network controller: Mellanox Technologies MT27500 > > >> > > > Family [ConnectX-3] > > >> > > > Subsystem: Mellanox Technologies Device 0051 > > >> > > > > > >> > > > Latest OFED driver: > > >> > > > > > >> > > > [root@ovirt-hv2 ~]# /etc/init.d/openibd status > > >> > > > > > >> > > > HCA driver loaded > > >> > > > > > >> > > > Configured IPoIB devices: > > >> > > > ib0 > > >> > > > > > >> > > > Currently active IPoIB devices: > > >> > > > ib0 > > >> > > > Configured Mellanox EN devices: > > >> > > > > > >> > > > Currently active Mellanox devices: > > >> > > > ib0 > > >> > > > > > >> > > > The following OFED modules are loaded: > > >> > > > > > >> > > > rdma_ucm > > >> > > > rdma_cm > > >> > > > ib_ipoib > > >> > > > mlx4_core > > >> > > > mlx4_ib > > >> > > > mlx4_en > > >> > > > mlx5_core > > >> > > > mlx5_ib > > >> > > > ib_uverbs > > >> > > > ib_umad > > >> > > > ib_ucm > > >> > > > ib_cm > > >> > > > ib_core > > >> > > > mlxfw > > >> > > > mlx5_fpga_tools > > >> > > > > > >> > > > I can add an IP to ib0 using "ip addr" though I need Network > > >> > > > Manager to work with ib0. > > >> > > > > > >> > > > > > >> > > > Thanks, > > >> > > > > > >> > > > Douglas Duckworth, MSc, LFCS > > >> > > > HPC System Administrator > > >> > > > Scientific Computing Unit > > >> > > > Weill Cornell Medicine > > >> > > > 1300 York - LC-502 > > >> > > > E: doug(a)med.cornell.edu > > >> > > > O: 212-746-6305 > > >> > > > F: 212-746-8690 > > >> > > > > >> > > > > >> > > >> > > > > >

2 1

Console to HostedEngine
by Daniel Menzel 24 Aug '18

24 Aug '18

Hi at all, we cannot access our hosted engine anymore. It started with and overfull /var due to a growing database. We access the engine via SSH and tried to fix that - but somehow we seem to have produced another problem on the SSH server itself. So unfortunately we can not login anymore. We then tried to access it via its host and a "hosted-engine --console" but ran into an internal error: cannot find character device <null> which I know from KVM. With other VMs I could follow the RedHat's advice to add a console (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/ht…) but although I edited the hosted engine's profile those changes weren't applied - and after an engine restart even deleted again. That kind of makes sense to me, but limits my options. So my question is: Is there any idea how can I access the the console with my current limitations to fix the SSH server's problems and then hopefully fix everything? Regards Daniel

2 4

Re: losing ib0 connection after activating host
by Douglas Duckworth 24 Aug '18

24 Aug '18

Hi Dominik Yes, the network-script was created by our Ansible role that deploys CentOS hosts. It pulls the IP from DNS then templates the script and copies to host. I will try this oVirt step then see if it works! Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Thu, Aug 23, 2018 at 11:09 AM, Dominik Holler <dholler(a)redhat.com> wrote: > Is ifcfg-ib0 created before adding the host? > Can ib0 be reconfigured using engine, e.g. by > "Compute > Hosts > hostx > Network Interfaces > Setup Host Networks"? > If this some kind of self-hosted engine? > > On Thu, 23 Aug 2018 09:30:59 -0400 > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > Here's a link to the files: > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__bit.ly_ > 2wjZ6Vo&d=DwICAg&c=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_ > 78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=Y25-OOvgu58jlC82- > fzBeNIpQ7ZscoHznffUhqE6EBM&s=QQXlC9Tisa60TvimyS3BnFDCaDF7VP > D8eCzT-Fke-p0&e= > > > > Thank you! > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York - LC-502 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > On Thu, Aug 23, 2018 at 6:51 AM, Dominik Holler <dholler(a)redhat.com> > > wrote: > > > > > Would you please share the vdsm.log and the supervdsm.log from this > > > host? > > > > > > On Wed, 22 Aug 2018 11:36:09 -0400 > > > Douglas Duckworth <dod2014(a)med.cornell.edu> wrote: > > > > > > > Hi > > > > > > > > I keep losing ib0 connection on hypervisor after adding host to > > > > engine. This makes the host not really work since NFS will be > > > > mounted over ib0. > > > > > > > > I don't really understand why this occurs. > > > > > > > > OS: > > > > > > > > [root@ovirt-hv2 ~]# cat /etc/redhat-release > > > > CentOS Linux release 7.5.1804 (Core) > > > > > > > > Here's the network script: > > > > > > > > [root@ovirt-hv2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib0 > > > > DEVICE=ib0 > > > > BOOTPROTO=static > > > > IPADDR=172.16.0.207 > > > > NETMASK=255.255.255.0 > > > > ONBOOT=yes > > > > ZONE=public > > > > > > > > When I try "ifup" > > > > > > > > [root@ovirt-hv2 ~]# ifup ib0 > > > > Error: Connection activation failed: No suitable device found for > > > > this connection. > > > > > > > > The error in syslog: > > > > > > > > Aug 22 11:31:50 ovirt-hv2 kernel: IPv4: martian source 172.16.0.87 > > > > from 172.16.0.49, on dev ib0 > > > > Aug 22 11:31:53 ovirt-hv2 NetworkManager[1070]: <info> > > > > [1534951913.7486] audit: op="connection-activate" > > > > uuid="2ab4abde-b8a5-6cbc-19b1-2bfb193e4e89" name="System ib0" > > > > result="fail" reason="No suitable device found for this > > > > connection. > > > > > > > > As you can see media state up: > > > > > > > > [root@ovirt-hv2 ~]# ip a > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > > > UNKNOWN group default qlen 1000 > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > > inet 127.0.0.1/8 scope host lo > > > > valid_lft forever preferred_lft forever > > > > 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master > > > > ovirtmgmt state UP group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > > > 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > > > state DOWN group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:82 brd ff:ff:ff:ff:ff:ff > > > > 4: p1p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > > > state DOWN group default qlen 1000 > > > > link/ether b4:96:91:13:ea:68 brd ff:ff:ff:ff:ff:ff > > > > 5: p1p2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq > > > > state DOWN group default qlen 1000 > > > > link/ether b4:96:91:13:ea:6a brd ff:ff:ff:ff:ff:ff > > > > 6: idrac: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > > > pfifo_fast state UNKNOWN group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:84 brd ff:ff:ff:ff:ff:ff > > > > inet 169.254.0.2/16 brd 169.254.255.255 scope global idrac > > > > valid_lft forever preferred_lft forever > > > > 7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state > > > > UP group default qlen 256 > > > > link/infiniband > > > > a0:00:02:08:fe:80:00:00:00:00:00:00:ec:0d:9a:03:00:1d:13:41 brd > > > > 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > > > > 8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state > > > > DOWN group default qlen 1000 > > > > link/ether 12:b4:30:22:39:5b brd ff:ff:ff:ff:ff:ff > > > > 9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > > > > group default qlen 1000 > > > > link/ether 3e:32:e6:66:98:49 brd ff:ff:ff:ff:ff:ff > > > > 25: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc > > > > noqueue state UP group default qlen 1000 > > > > link/ether 50:9a:4c:89:d3:81 brd ff:ff:ff:ff:ff:ff > > > > inet 10.0.0.183/16 brd 10.0.255.255 scope global ovirtmgmt > > > > valid_lft forever preferred_lft forever > > > > 26: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 > > > > qdisc noqueue master ovs-system state UNKNOWN group default qlen > > > > 1000 link/ether aa:32:82:1b:01:d9 brd ff:ff:ff:ff:ff:ff > > > > 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state > > > > DOWN group default qlen 1000 > > > > link/ether 32:ff:5d:b8:c2:b4 brd ff:ff:ff:ff:ff:ff > > > > > > > > The card is FDR: > > > > > > > > [root@ovirt-hv2 ~]# lspci -v | grep Mellanox > > > > 01:00.0 Network controller: Mellanox Technologies MT27500 Family > > > > [ConnectX-3] > > > > Subsystem: Mellanox Technologies Device 0051 > > > > > > > > Latest OFED driver: > > > > > > > > [root@ovirt-hv2 ~]# /etc/init.d/openibd status > > > > > > > > HCA driver loaded > > > > > > > > Configured IPoIB devices: > > > > ib0 > > > > > > > > Currently active IPoIB devices: > > > > ib0 > > > > Configured Mellanox EN devices: > > > > > > > > Currently active Mellanox devices: > > > > ib0 > > > > > > > > The following OFED modules are loaded: > > > > > > > > rdma_ucm > > > > rdma_cm > > > > ib_ipoib > > > > mlx4_core > > > > mlx4_ib > > > > mlx4_en > > > > mlx5_core > > > > mlx5_ib > > > > ib_uverbs > > > > ib_umad > > > > ib_ucm > > > > ib_cm > > > > ib_core > > > > mlxfw > > > > mlx5_fpga_tools > > > > > > > > I can add an IP to ib0 using "ip addr" though I need Network > > > > Manager to work with ib0. > > > > > > > > > > > > Thanks, > > > > > > > > Douglas Duckworth, MSc, LFCS > > > > HPC System Administrator > > > > Scientific Computing Unit > > > > Weill Cornell Medicine > > > > 1300 York - LC-502 > > > > E: doug(a)med.cornell.edu > > > > O: 212-746-6305 > > > > F: 212-746-8690 > > > > > > > >

2 2

Re: cannot migrate hosted engine to sandybridge hypervisor
by Douglas Duckworth 24 Aug '18

24 Aug '18

Thanks! No big deal as it's now working. On Fri, Aug 24, 2018, 9:04 AM Simone Tiraboschi <stirabos(a)redhat.com> wrote: > > > On Fri, Aug 24, 2018 at 3:06 AM Douglas Duckworth <dod2014(a)med.cornell.edu> > wrote: > >> Rebooted the hosted engine VM >> >> Migration now works >> >> Guess that's needed after changing cluster CPU type >> > > Yes, exactly: we already have an open bug about that: > https://bugzilla.redhat.com/show_bug.cgi?id=1585986 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.redhat.com_sh…> > > >> >> Thanks, >> >> Douglas Duckworth, MSc, LFCS >> HPC System Administrator >> Scientific Computing Unit >> Weill Cornell Medicine >> 1300 York - LC-502 >> E: doug(a)med.cornell.edu >> O: 212-746-6305 >> F: 212-746-8690 >> >> >> On Thu, Aug 23, 2018 at 8:46 PM, Douglas Duckworth < >> dod2014(a)med.cornell.edu> wrote: >> >>> Hi >>> >>> I am unable to migrate to hypervisor that's sandy bridge CPU. >>> >>> The cluster supports CPUs that old. >>> >>> This is the message on host when migration fails: >>> >>> Aug 23 20:40:14 lyon vdsm[9648]: WARN Worker blocked: <Worker >>> name=periodic/1 running <Task <Operation >>> action=<vdsm.virt.sampling.HostMonitor object at 0x7f2f8054b310> at >>> 0x7f2f8054b350> timeout=15, duration=15 at 0x7f2f8048d290> task#=79 at >>> 0x7f2f80540490>, traceback:#012File: "/usr/lib64/python2.7/threading.py", >>> line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: >>> "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 >>> self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in >>> run#012 self.__target(*self.__args, **self.__kwargs)#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in >>> run#012 ret = func(*args, **kwargs)#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 >>> self._execute_task()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in >>> _execute_task#012 task()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in >>> __call__#012 self._callable()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in >>> __call__#012 self._func()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in >>> __call__#012 stats = hostapi.get_stats(self._cif, >>> self._samples.stats())#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in >>> get_stats#012 ret['haStats'] = _getHaInfo()#012File: >>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in >>> _getHaInfo#012 stats = instance.get_all_stats()#012File: >>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", >>> line 94, in get_all_stats#012 stats = >>> broker.get_stats_from_storage()#012File: >>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", >>> line 135, in get_stats_from_storage#012 result = >>> self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line >>> 1233, in __call__#012 return self.__send(self.__name, args)#012File: >>> "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 >>> verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line >>> 1273, in request#012 return self.single_request(host, handler, >>> request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line >>> 1303, in single_request#012 response = >>> h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", >>> line 1113, in getresponse#012 response.begin()#012File: >>> "/usr/lib64/python2.7/httplib.py", line 444, in begin#012 version, status, >>> reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", >>> line 400, in _read_status#012 line = self.fp.readline(_MAXLINE + >>> 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012 >>> data = self._sock.recv(self._rbufsize) >>> Aug 23 20:43:45 lyon vdsm[9648]: WARN ping was deprecated in favor of >>> ping2 and confirmConnectivity >>> Aug 23 20:43:45 lyon vdsm[9648]: WARN Attempting to add an existing net >>> user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b >>> Aug 23 20:43:46 lyon lldpad: recvfrom(Event interface): No buffer space >>> available >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking >>> state >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled >>> state >>> Aug 23 20:43:46 lyon kernel: device vnet0 entered promiscuous mode >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking >>> state >>> Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered forwarding >>> state >>> Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4326] >>> manager: (vnet0): new Tun device >>> (/org/freedesktop/NetworkManager/Devices/57) >>> Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4336] >>> device (vnet0): state change: unmanaged -> unavailable (reason >>> 'connection-assumed', sys-iface-state: 'external') >>> Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4341] >>> device (vnet0): state change: unavailable -> disconnected (reason 'none', >>> sys-iface-state: 'external') >>> Aug 23 20:43:46 lyon dbus[1396]: [system] Activating via systemd: >>> service name='org.freedesktop.machine1' >>> unit='dbus-org.freedesktop.machine1.service' >>> Aug 23 20:43:46 lyon systemd: Cannot add dependency job for unit >>> lvm2-lvmetad.socket, ignoring: Unit is masked. >>> Aug 23 20:43:46 lyon systemd: Starting Virtual Machine and Container >>> Registration Service... >>> Aug 23 20:43:46 lyon dbus[1396]: [system] Successfully activated service >>> 'org.freedesktop.machine1' >>> Aug 23 20:43:46 lyon systemd: Started Virtual Machine and Container >>> Registration Service. >>> Aug 23 20:43:46 lyon systemd-machined: New machine qemu-2-HostedEngine. >>> Aug 23 20:43:46 lyon systemd: Started Virtual Machine >>> qemu-2-HostedEngine. >>> Aug 23 20:43:46 lyon systemd: Starting Virtual Machine >>> qemu-2-HostedEngine. >>> Aug 23 20:43:46 lyon kvm: 1 guest now active >>> Aug 23 20:43:46 lyon libvirtd: 2018-08-24 00:43:46.940+0000: 9195: error >>> : * virCPUx86UpdateLive:2742 : operation failed: guest CPU doesn't >>> match specification: missing features: >>> fma,movbe,f16c,rdrand,fsgsbase,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,mpx,rdseed,adx,smap,xsavec,xgetbv1,abm,3dnowprefetch* >>> Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled >>> state >>> Aug 23 20:43:47 lyon kernel: device vnet0 left promiscuous mode >>> Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled >>> state >>> Aug 23 20:43:47 lyon avahi-daemon[1300]: Withdrawing workstation service >>> for vnet0. >>> Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] >>> device (vnet0): state change: disconnected -> unmanaged (reason >>> 'unmanaged', sys-iface-state: 'removed') >>> Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] >>> device (vnet0): released from master device ovirtmgmt >>> Aug 23 20:43:47 lyon lldpad: recvfrom(Event interface): No buffer space >>> available >>> Aug 23 20:43:47 lyon kvm: 0 guests now active >>> Aug 23 20:43:47 lyon systemd-machined: Machine qemu-2-HostedEngine >>> terminated. >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN File: >>> /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.ovirt-guest-agent.0 >>> already removed >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non >>> existing network: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non >>> existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b >>> Aug 23 20:43:47 lyon vdsm[9648]: WARN File: >>> /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.org.qemu.guest_agent.0 >>> already removed >>> Aug 23 20:43:51 lyon lldpad: setsockopt nearest_bridge: Invalid argument >>> Aug 23 20:43:52 lyon lldpad: setsockopt nearest_bridge: Invalid argument >>> >>> Can I fix this somehow? >>> >>> Thanks, >>> >>> Douglas Duckworth, MSc, LFCS >>> HPC System Administrator >>> Scientific Computing Unit >>> Weill Cornell Medicine >>> 1300 York - LC-502 >>> E: doug(a)med.cornell.edu >>> O: 212-746-6305 >>> F: 212-746-8690 >>> >>> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_site_pri…> >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ovirt.org_communit…> >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FBENIU2TRZSA2… >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ovirt.org_archiv…> >> >

1 0

cannot migrate hosted engine to sandybridge hypervisor
by Douglas Duckworth 24 Aug '18

24 Aug '18

Hi I am unable to migrate to hypervisor that's sandy bridge CPU. The cluster supports CPUs that old. This is the message on host when migration fails: Aug 23 20:40:14 lyon vdsm[9648]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f2f8054b310> at 0x7f2f8054b350> timeout=15, duration=15 at 0x7f2f8048d290> task#=79 at 0x7f2f80540490>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in __call__#012 self._func()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in __call__#012 stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats#012 ret['haStats'] = _getHaInfo()#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo#012 stats = instance.get_all_stats()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats#012 stats = broker.get_stats_from_storage()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage#012 result = self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012 response = h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse#012 response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444, in begin#012 version, status, reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012 line = self.fp.readline(_MAXLINE + 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012 data = self._sock.recv(self._rbufsize) Aug 23 20:43:45 lyon vdsm[9648]: WARN ping was deprecated in favor of ping2 and confirmConnectivity Aug 23 20:43:45 lyon vdsm[9648]: WARN Attempting to add an existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b Aug 23 20:43:46 lyon lldpad: recvfrom(Event interface): No buffer space available Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking state Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state Aug 23 20:43:46 lyon kernel: device vnet0 entered promiscuous mode Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking state Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4326] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/57) Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4336] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Aug 23 20:43:46 lyon NetworkManager[1658]: <info> [1535071426.4341] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Aug 23 20:43:46 lyon dbus[1396]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service' Aug 23 20:43:46 lyon systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 23 20:43:46 lyon systemd: Starting Virtual Machine and Container Registration Service... Aug 23 20:43:46 lyon dbus[1396]: [system] Successfully activated service 'org.freedesktop.machine1' Aug 23 20:43:46 lyon systemd: Started Virtual Machine and Container Registration Service. Aug 23 20:43:46 lyon systemd-machined: New machine qemu-2-HostedEngine. Aug 23 20:43:46 lyon systemd: Started Virtual Machine qemu-2-HostedEngine. Aug 23 20:43:46 lyon systemd: Starting Virtual Machine qemu-2-HostedEngine. Aug 23 20:43:46 lyon kvm: 1 guest now active Aug 23 20:43:46 lyon libvirtd: 2018-08-24 00:43:46.940+0000: 9195: error : *virCPUx86UpdateLive:2742 : operation failed: guest CPU doesn't match specification: missing features: fma,movbe,f16c,rdrand,fsgsbase,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,mpx,rdseed,adx,smap,xsavec,xgetbv1,abm,3dnowprefetch* Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state Aug 23 20:43:47 lyon kernel: device vnet0 left promiscuous mode Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state Aug 23 20:43:47 lyon avahi-daemon[1300]: Withdrawing workstation service for vnet0. Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Aug 23 20:43:47 lyon NetworkManager[1658]: <info> [1535071427.2753] device (vnet0): released from master device ovirtmgmt Aug 23 20:43:47 lyon lldpad: recvfrom(Event interface): No buffer space available Aug 23 20:43:47 lyon kvm: 0 guests now active Aug 23 20:43:47 lyon systemd-machined: Machine qemu-2-HostedEngine terminated. Aug 23 20:43:47 lyon vdsm[9648]: WARN File: /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.ovirt-guest-agent.0 already removed Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non existing network: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b Aug 23 20:43:47 lyon vdsm[9648]: WARN File: /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.org.qemu.guest_agent.0 already removed Aug 23 20:43:51 lyon lldpad: setsockopt nearest_bridge: Invalid argument Aug 23 20:43:52 lyon lldpad: setsockopt nearest_bridge: Invalid argument Can I fix this somehow? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York - LC-502 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690

2 2

Upgrading librbd1 and librados2 for oVirt 4.2.x
by andreas.elvers+ovirtforum＠solutions.work 24 Aug '18

24 Aug '18

Hallo Group ! I am in need of upgrading librbd1 and librados2 for a oVirt 4.2.x cluster. The cluster was installed via node ng. Taking the repository for Ceph Mimic or Luminous will end in a dependency problem because liburcu-cds.so.1 is already installed as a more recent version provided by the oVirt repository. A hint on how to get a more recent versions of the above libraries would be very much appreciated. Thanks, - Andreas

1 1