- Users - oVirt List Archives

ovirt4 api search storage domain
by David David 04 Sep '18

04 Sep '18

hi need to lookup storage name by vm disk id. how to do it in python or in java? thanks

2 1

Wrong network threshold limit warnings on 4.2.5
by Florian Schmid 04 Sep '18

04 Sep '18

Good morning, since we have upgraded to version 4.2.5, we get a lot of warnings about network interface exceeded defined threshold limits. For example: Aug 31, 2018, 7:54:05 AM Host xxx has network interface which exceeded the defined threshold [95%] (enp9s0.80: transmit rate[100%], receive rate [12%]) This is a 10 Gbit interface and on our monitoring software, which is getting network statistics every 10s, the bandwidth of TX was 150 Mbit maximum at this time, so far away from being 100%. Could it be, that the engine detected the wrong interface speed or there is a calculation error? In the engine for this host, I have 10000 Mbps for all interfaces. I have checked now all those warnings on our different hosts and they happen every time, we go over 100 Mbit and this is for sure quite often... Can I maybe disable these warnings, because we have it anyway in our monitoring software? If you need any logs, please ask. BR Florian Schmid

5 10

Bricks do not appear in oVirt Manager UI under Hosts unless manually created on nodes.
by payden.pringle+ovirtlist＠gmail.com 03 Sep '18

03 Sep '18

I have a CentOS 7.5 system running the oVirt-Engine package as the manager for three oVirt Nodes (version 4.2). Two of the nodes have a 200GB disk and one has a 2GB disk. I am attempting to create a Gluster Volume that replicates between the two 200GB disks with the 2GB as Arbiter. When I create the bricks by going to Hosts > host1 > Storage Devices > Create Brick (with the 200GB device selected), it succeeds, but no bricks appear under the Bricks tab for the Host. Doing the same for host2 and host3 results in the same problem. If I run the following command on a host `gluster volume create test-volume replica 3 arbiter 1 192.168.5.{50,60,40}:/gluster-bricks/200GBbrick/test`, it creates the volume correctly and the bricks then appear as they should. However, then these Bricks are labeled as "Down" and when I go to Storage > Volumes, I get this error: ``` Uncaught exception occurred. Please try reloading the page. Details: (TypeError) : Cannot read property 'a' of null Please have your administrator check the UI logs ``` The Volume is also shown as "Down". SELinux is enabled on my CentOS 7.5 system. I'm not really sure what the correct method is to create bricks then a Volume in Gluster through the oVirt Management UI, but I've followed the guides and this is where it has got me.

2 1

oVirt Node 4.2.3.1. to 4.2.5 upgrade trouble, log attached
by Matt Simonsen 03 Sep '18

03 Sep '18

Hello all, I've emailed about similar trouble with an oVirt Node upgrade using the ISO install. I've attached the /tmp/imgbased.log file in hopes it will help give a clue as to why the trouble. Since these use NFS storage I can rebuild, but would like to know, ideally, what caused the upgrade to break. Truthfully following the install, I don't think I have done *that* much to these systems, so I'm not sure what would have caused the problem. I have done several successful upgrades in the past and most of my standalone systems have been working great. I've been really happy with oVirt, so kudos to the team. Thanks for any help, Matt

5 13

the issue of starting a VM
by hhz711＠126.com 03 Sep '18

03 Sep '18

hello: I created a VM on node1, but when I power on the vm and try to boot from the CD-ROM with ISO, but it's stucked . the boot window only show below info: "seaBIOS ( version xxxxxx） Machine UUID xxxxxxxx " Did you guys have met this before ?

1 0

Gluster clients intermittently hang until first gluster server in a Replica 1 Arbiter 1 cluster is rebooted, server error: 0-management: Unlocking failed & client error: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2131 sent = <datestamp>. timeout = 1800
by Sam McLeod 03 Sep '18

03 Sep '18

We've got an odd problem where clients are blocked from writing to Gluster volumes until the first node of the Gluster cluster is rebooted. I suspect I've either configured something incorrectly with the arbiter / replica configuration of the volumes, or there is some sort of bug in the gluster client-server connection that we're triggering. I was wondering if anyone has seen this or could point me in the right direction? Environment: Typology: 3 node cluster, replica 2, arbiter 1 (third node is metadata only). Version: Client and Servers both running 4.1.3, both on CentOS 7, kernel 4.18.x, (Xen) VMs with relatively fast networked SSD storage backing them, XFS. Client: Native Gluster FUSE client mounting via the kubernetes provider Problem: Seemingly randomly some clients will be blocked / are unable to write to what should be a highly available gluster volume. The client gluster logs show it failing to do new file operations across various volumes and all three nodes of the gluster. The server gluster (or OS) logs do not show any warnings or errors. The client recovers and is able to write to volumes again after the first node of the gluster cluster is rebooted. Until the first node of the gluster cluster is rebooted, the client fails to write to the volume that is (or should be) available on the second node (a replica) and third node (an arbiter only node). What 'fixes' the issue: Although the clients (kubernetes hosts) connect to all 3 nodes of the Gluster cluster - restarting the first gluster node always unblocks the IO and allows the client to continue writing. Stopping and starting the glusterd service on the gluster server is not enough to fix the issue, nor is restarting its networking. This suggests to me that the volume is unavailable for writing for some reason and restarting the first node in the cluster either clears some sort of TCP sessions between the client-server or between the server-server replication. Expected behaviour: If the first gluster node / server had failed or was blocked from performing operations for some reason (which it doesn't seem it is), I'd expect the clients to access data from the second gluster node and write metadata to the third gluster node as well as it's an arbiter / metadata only node. If for some reason the a gluster node was not able to serve connections to clients, I'd expect to see errors in the volume, glusterd or brick log files (there are none on the first gluster node). If the first gluster node was for some reason blocking IO on a volume, I'd expect that node either to show as unhealthy or unavailable in the gluster peer status or gluster volume status. Client gluster errors: staging_static in this example is a volume name. You can see the client trying to connect to the second and third nodes of the gluster cluster and failing (unsure as to why?) The server side logs on the first gluster node do not show any errors or problems, but the second / third node show errors in the glusterd.log when trying to 'unlock' the 0-management volume on the first node. On a gluster client (a kubernetes host using the kubernetes connector which uses the native fuse client) when its blocked from writing but the gluster appears healthy (other than the errors mentioned later): [2018-09-02 15:33:22.750874] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1cce sent = 2018-09-02 15:03:22.417773. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 15:33:22.750989] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 16:03:23.097905] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2e21 sent = 2018-09-02 15:33:22.765751. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 16:03:23.097988] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 16:33:23.439172] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1d4b sent = 2018-09-02 16:03:23.098133. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 16:33:23.439282] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 17:03:23.786858] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2ee7 sent = 2018-09-02 16:33:23.455171. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 17:03:23.786971] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 17:33:24.160607] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1dc8 sent = 2018-09-02 17:03:23.787120. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 17:33:24.160720] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 18:03:24.505092] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2faf sent = 2018-09-02 17:33:24.173153. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 18:03:24.505185] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 18:33:24.841248] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1e45 sent = 2018-09-02 18:03:24.505328. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 18:33:24.841311] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 19:03:25.204711] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3074 sent = 2018-09-02 18:33:24.855372. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 19:03:25.204784] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 19:33:25.533545] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1ec2 sent = 2018-09-02 19:03:25.204977. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 19:33:25.533611] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 20:03:25.877020] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3138 sent = 2018-09-02 19:33:25.545921. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 20:03:25.877098] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 20:33:26.217858] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1f3e sent = 2018-09-02 20:03:25.877264. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 20:33:26.217973] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 21:03:26.588237] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x31ff sent = 2018-09-02 20:33:26.233010. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 21:03:26.588316] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 21:33:26.912334] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x1fbb sent = 2018-09-02 21:03:26.588456. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 21:33:26.912449] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 22:03:37.258915] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x32c5 sent = 2018-09-02 21:33:32.091009. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 22:03:37.259000] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 22:33:37.615497] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2039 sent = 2018-09-02 22:03:37.259147. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 22:33:37.615574] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-02 23:03:37.940969] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3386 sent = 2018-09-02 22:33:37.629655. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-02 23:03:37.941049] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-02 23:33:38.270998] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x20b5 sent = 2018-09-02 23:03:37.941199. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-02 23:33:38.271078] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-03 00:03:38.607186] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x3447 sent = 2018-09-02 23:33:38.285899. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-03 00:03:38.607263] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-03 00:33:38.934385] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x2131 sent = 2018-09-03 00:03:38.607410. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-03 00:33:38.934479] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] [2018-09-03 01:03:39.256842] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-1: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x350c sent = 2018-09-03 00:33:38.948570. timeout = 1800 for <ip of second gluster node>:49154 [2018-09-03 01:03:39.256972] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-1: remote operation failed [Transport endpoint is not connected] [2018-09-03 01:33:39.614402] E [rpc-clnt.c:184:call_bail] 0-staging_static-client-2: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x21ae sent = 2018-09-03 01:03:39.258166. timeout = 1800 for <ip of third gluster node>:49154 [2018-09-03 01:33:39.614483] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-staging_static-client-2: remote operation failed [Transport endpoint is not connected] On the second gluster server: We are seeing the following error in the glusterd.log file when the client is blocked from writing the volume, I think this is probably the most important information about the error and suggests a problem with the first node but doesn't explain the client behaviour: [2018-09-02 08:31:03.902272] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on <FQDN of the first gluster node>. Please check log file for details. [2018-09-02 08:31:03.902477] E [MSGID: 106151] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s) Note in the above error: 1. I'm not sure which log to check (there doesn't seem to be a management brick / brick log)? 2. If there's a problem with the first node, why isn't it rejected from the gluster / taken offline / the health of the peers or volume list degraded? 3. Why does the client fail to write to the volume rather than (I'm assuming) trying the second (or third I guess) node to write to the volume? We are also seeing the following errors repeated a lot in the logs, both when the volumes are working and when there's an issue in the brick log (/var/log/glusterfs/bricks/mnt-gluster-storage-staging_static.log): [2018-09-03 01:58:35.128923] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.128957] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3d60, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.128983] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.129016] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3e2a, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.129042] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.129077] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3ef6, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.129149] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed [2018-09-03 01:58:35.129191] E [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x3fc6, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 29) to rpc-transport (tcp.staging_static-server) [2018-09-03 01:58:35.129219] E [server.c:137:server_submit_reply] (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x1fd14) [0x7f8470319d14] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x5f24a) [0x7f846bdde24a] -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce) [0x7f846bd89fce] ) 0-: Reply submission failed Gluster volume information: # gluster volume info staging_static Volume Name: staging_static Type: Replicate Volume ID: 7f3b8e91-afea-4fc6-be83-3399a089b6f3 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: <first gluster node.fqdn>:/mnt/gluster-storage/staging_static Brick2: <second gluster node.fqdn>:/mnt/gluster-storage/staging_static Brick3: <third gluster node.fqdn>:/mnt/gluster-storage/staging_static (arbiter) Options Reconfigured: storage.fips-mode-rchecksum: true cluster.self-heal-window-size: 16 cluster.shd-wait-qlength: 4096 cluster.shd-max-threads: 8 performance.cache-min-file-size: 2KB performance.rda-cache-limit: 1GB network.inode-lru-limit: 50000 server.outstanding-rpc-limit: 256 transport.listen-backlog: 2048 performance.write-behind-window-size: 512MB performance.stat-prefetch: true performance.io-thread-count: 16 performance.client-io-threads: true performance.cache-size: 1GB performance.cache-refresh-timeout: 60 performance.cache-invalidation: true cluster.use-compound-fops: true cluster.readdir-optimize: true cluster.lookup-optimize: true cluster.favorite-child-policy: size cluster.eager-lock: true client.event-threads: 4 nfs.disable: on transport.address-family: inet diagnostics.brick-log-level: ERROR diagnostics.client-log-level: ERROR features.cache-invalidation-timeout: 300 features.cache-invalidation: true network.ping-timeout: 15 performance.cache-max-file-size: 3MB performance.md-cache-timeout: 300 server.event-threads: 4 Thanks in advance, -- Sam McLeod (protoporpoise on IRC) https://smcleod.net https://twitter.com/s_mcleod Words are my own opinions and do not necessarily represent those of my employer or partners.

1 1

Windows vm and cpu type
by carl langlois 31 Aug '18

31 Aug '18

Hi all, I am trying to create a windows VM on my cluster that is set with cpu type to Nehalem. The probleme is that ovirt tell me that the guest os does not support this cpu.. What is the best ways to fix it without impacting my 41 running vms?. Thanks Regards

2 3

Re: ovirt-engine-extension-aaa-ldap-setup credentials invalid
by Douglas Duckworth 31 Aug '18

31 Aug '18

Thanks! I set that after completing setup though still cannot login: 2018-08-30 10:42:46,470-04 ERROR [org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default task-5) [] Cannot authenticate user 'me@scu' connecting from '10.0.0.136': Unable to log in. Verify your login information or contact the system administrator. How can I get get more verbose logging? Thanks, Douglas Duckworth, MSc, LFCS HPC System Administrator Scientific Computing Unit Weill Cornell Medicine 1300 York Avenue New York, NY 10065 E: doug(a)med.cornell.edu O: 212-746-6305 F: 212-746-8690 On Thu, Aug 30, 2018 at 8:36 AM, Ondra Machacek <omachace(a)redhat.com> wrote: > On 08/29/2018 08:04 PM, Douglas Duckworth wrote: > > Hi > > > > I am not able to finish ovirt-engine-extension-aaa-ldap-setup due to the > > error "credentials invalid." > > > > My password's correct. Our directory server's OpenLDAP standard schema. > > Our accounts are under OU=People. We are binding with a service account > > that has access to the entire DN. > > > > Any way to discern more information from attached logs? > > As you can see you get: > > 2018-08-29 13:47:44,129-04 WARNING Exception: anonymous bind disallowed > > More info how to solve this issue is here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists. > ovirt.org_pipermail_users_2018-2DFebruary_086924.html&d=DwICaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s= > GKdSUL68LUZfQo0jXz6oizzkl38dkjJBqsxlNjlNhg0&e= > > > > > > > > > Thanks, > > > > Douglas Duckworth, MSc, LFCS > > HPC System Administrator > > Scientific Computing Unit > > Weill Cornell Medicine > > 1300 York Avenue > > New York, NY 10065 > > E: doug(a)med.cornell.edu > > O: 212-746-6305 > > F: 212-746-8690 > > > > > > > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://urldefense.proofpoint. > com/v2/url?u=https-3A__www.ovirt.org_site_privacy-2Dpolicy_&d=DwICaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s= > EMrnI4emKvoDHpfvwPDhoIiX_UZd3_cUeYWCl26APHk&e= > > oVirt Code of Conduct: https://urldefense.proofpoint. > com/v2/url?u=https-3A__www.ovirt.org_community_about_ > community-2Dguidelines_&d=DwICaQ&c=lb62iw4YL4RFalcE2hQUQealT9- > RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_ > P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s=K5tHvffPexNoF9tOFEFopJcv4ROA1s > JUfLr9dvAglXM&e= > > List Archives: https://urldefense.proofpoint. > com/v2/url?u=https-3A__lists.ovirt.org_archives_list_users- > 40ovirt.org_message_ROWSZGMEMSYKUWEIDAKZMSVTKHPPOEDN_&d=DwICaQ&c= > lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s&r=2Fzhh_78OGspKQpl_e- > CbhH6xUjnRkaqPFUS2wTJ2cw&m=4tJ1lC_P1F3IAyLlVozg5gbalhcKCbsX2fQO69cYaWY&s= > lyUWPBTItHvb0nXgveWBdPUGNTyuAdUx4OnaME4dOrY&e= > > >

2 1

Disk Statistics error
by Hari Prasanth Loganathan 31 Aug '18

31 Aug '18

Hi Team, I try to get the disk statistics using the following *Url: * * /ovirt-engine/api/disks?follow=statistics* and I get the following error in oVirt, 2018-08-17 15:34:24,663+08 ERROR [io.undertow.request] (default task-815) UT005023: Exception handling request to /ovirt-engine/api/v4/disks: java.lang.RuntimeException: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: Problem following 'statistics' link in Disks entity. at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl( RequestDispatcherImpl.java:245) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup( RequestDispatcherImpl.java:147) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at io.undertow.servlet.spec.RequestDispatcherImpl.forward( RequestDispatcherImpl.java:111) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.api.restapi.invocation.VersionFilter. doFilter(VersionFilter.java:180) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.VersionFilter. doFilter(VersionFilter.java:98) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.api.restapi.invocation.CurrentFilter. doFilter(CurrentFilter.java:117) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.invocation.CurrentFilter. doFilter(CurrentFilter.java:72) [restapi-jaxrs.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter. doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter. doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter. doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter. doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.9.Final.jar:2.0.9.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) [undertow-servlet-2.0.9.Final. jar:2.0.9.Final] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter. doFilter(SessionValidationFilter.java:59) [aaa.jar:] Caused by: java.lang.IllegalStateException: Problem following 'statistics' link in Disks entity. at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. fetchData(LinkFollower.java:185) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. followLink(LinkFollower.java:144) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. followLinks(LinkFollower.java:132) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. followLinks(LinkFollower.java:53) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BaseBackendResource. follow(BaseBackendResource.java:412) [restapi-jaxrs.jar:] at org.ovirt.engine.api.resource.DisksResource.doList(DisksResource.java:222) [restapi-definition.jar:] at sun.reflect.GeneratedMethodAccessor407.invoke(Unknown Source) [:1.8.0_181] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker. internalInvokeOnTarget(ResourceMethodInvoker.java:510) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker. invokeOnTargetAfterFilter(ResourceMethodInvoker.java:401) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$ invokeOnTarget$0(ResourceMethodInvoker.java:365) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.interception. PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:361) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget( ResourceMethodInvoker.java:367) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( ResourceMethodInvoker.java:339) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker. invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-20-3.5.1.Final.jar:3.5.1.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke( ResourceLocatorInvoker.java:100) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke( SynchronousDispatcher.java:441) [resteasy-jaxrs-20-3.5.1. Final.jar:3.5.1.Final] ... 96 more Caused by: java.lang.IllegalStateException: Problem fetching 'statistics' from Disk at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. fetchData(LinkFollower.java:219) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.utils.LinkFollower. fetchData(LinkFollower.java:182) [restapi-jaxrs.jar:] ... 114 more Caused by: java.lang.NullPointerException Any thoughts on this? Thanks, Hari -- DISCLAIMER - *MSysTechnologies LLC* This email message, contents and its attachments may contain confidential, proprietary or legally privileged information and is intended solely for the use of the individual or entity to whom it is actually intended. If you have erroneously received this message, please permanently delete it immediately and notify the sender. If you are not the intended recipient of the email message,you are notified strictly not to disseminate,distribute or copy this e-mail.E-mail transmission cannot be guaranteed to be secure or error-free as Information could be intercepted, corrupted, lost, destroyed, incomplete or contain viruses and MSysTechnologies LLC accepts no liability for the contents and integrity of this mail or for any damage caused by the limitations of the e-mail transmission.

2 6

Possible vdsmd issues - " WARN Worker blocked"
by Johnson, Tim 31 Aug '18

31 Aug '18

Hi all, We have had a bit of a strange issue with the starting of the self-hosted engine due to the vdsm and ovirt-ha processes having errors and not able to be started. Running Ovirt version 4.2 with Gluster mounts (not hyper converged) gluster 3.12.13 replicated volumes with arbiter brick. The mounts for the storage are mounted on all the Ovirt hosts and able to read/touch files on the mounts. Unfortunately we can not pin a specific timeframe when this started happening. Meaning we don’t know if it started after a specific update or any other event. The vdsmd is throwing errors (bottom – with version numbers) when trying to check the status of the self -hosted engine getting : # hosted-engine --vm-status The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. Thanks for any assistance. systemctl status ovirt-ha-agent -l ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Thu 2018-08-30 13:24:40 CDT; 2s ago Process: 17135 ExecStart=/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent (code=exited, status=157) Main PID: 17135 (code=exited, status=157) systemctl status ovirt-ha-broker -l ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2018-08-30 13:10:35 CDT; 14min ago Main PID: 14863 (ovirt-ha-broker) Tasks: 4 CGroup: /system.slice/ovirt-ha-broker.service └─14863 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker Aug 30 13:10:35 rrb-vmhost1.rrc.local systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Aug 30 13:10:35 rrb-vmhost1.rrc.local systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker... Aug 30 13:14:44 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:14:44 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.Action.start_domain_monitor ERROR Error in RPC call: Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.Action.start_domain_monitor ERROR Error in RPC call: Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout during domain acquisition Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Failed to update state. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 82, in run if (self._status_broker._inquire_whiteboard_lock() or File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 190, in _inquire_whiteboard_lock self.host_id, self._lease_file) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 128, in host_id raise ex.HostIdNotLockedError("Host id is not set") HostIdNotLockedError: Host id is not set Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Trying to restart the broker Aug 30 13:19:42 rrb-vmhost1.rrc.local python[14863]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Aug 30 13:19:48 rrb-vmhost1.rrc.local python[14863]: communication with ABRT daemon failed: timed out glusterfs-api-3.12.13-1.el7.x86_64 glusterfs-server-3.12.13-1.el7.x86_64 glusterfs-fuse-3.12.13-1.el7.x86_64 vdsm-gluster-4.20.35-1.el7.x86_64 centos-release-gluster312-1.0-2.el7.centos.noarch glusterfs-client-xlators-3.12.13-1.el7.x86_64 glusterfs-geo-replication-3.12.13-1.el7.x86_64 glusterfs-libs-3.12.13-1.el7.x86_64 glusterfs-cli-3.12.13-1.el7.x86_64 glusterfs-rdma-3.12.13-1.el7.x86_64 glusterfs-events-3.12.13-1.el7.x86_64 glusterfs-3.12.13-1.el7.x86_64 libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64 python2-gluster-3.12.13-1.el7.x86_64 glusterfs-gnfs-3.12.13-1.el7.x86_64 ovirt-setup-lib-1.1.4-1.el7.centos.noarch ovirt-iso-uploader-4.2.0-1.el7.centos.noarch ovirt-engine-setup-base-4.2.5.3-1.el7.noarch ovirt-host-dependencies-4.2.3-1.el7.x86_64 ovirt-guest-agent-common-1.0.14-1.el7.noarch python-ovirt-engine-sdk4-4.2.7-2.el7.x86_64 ovirt-engine-setup-plugin-ovirt-engine-common-4.2.5.3-1.el7.noarch ovirt-vmconsole-1.0.5-4.el7.centos.noarch ovirt-engine-tools-backup-4.2.5.3-1.el7.noarch ovirt-guest-agent-windows-1.0.14-1.el7.centos.noarch cockpit-ovirt-dashboard-0.11.31-1.el7.noarch ovirt-host-deploy-1.7.4-1.el7.noarch ovirt-engine-lib-4.2.5.3-1.el7.noarch ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-hosted-engine-ha-2.2.16-1.el7.noarch ovirt-guest-tools-iso-4.2-1.el7.centos.noarch ovirt-imageio-common-1.4.2-0.el7.noarch ovirt-hosted-engine-setup-2.2.25-1.el7.noarch ovirt-imageio-daemon-1.4.2-0.el7.noarch ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch ovirt-provider-ovn-driver-1.2.13-1.el7.noarch ovirt-host-4.2.3-1.el7.x86_64 vdsm-jsonrpc-4.20.35-1.el7.noarch vdsm-hook-ethtool-options-4.20.35-1.el7.noarch vdsm-gluster-4.20.35-1.el7.x86_64 vdsm-api-4.20.35-1.el7.noarch vdsm-yajsonrpc-4.20.35-1.el7.noarch vdsm-client-4.20.35-1.el7.noarch vdsm-hook-fcoe-4.20.35-1.el7.noarch vdsm-network-4.20.35-1.el7.x86_64 vdsm-hook-vmfex-dev-4.20.35-1.el7.noarch vdsm-hook-vhostmd-4.20.35-1.el7.noarch vdsm-http-4.20.35-1.el7.noarch vdsm-hook-openstacknet-4.20.35-1.el7.noarch vdsm-python-4.20.35-1.el7.noarch vdsm-hook-vfio-mdev-4.20.35-1.el7.noarch vdsm-common-4.20.35-1.el7.noarch vdsm-4.20.35-1.el7.x86_64 Aug 30 13:02:42 rrb-vmhost1.rrc.local vdsm[6792]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f713c0b0fd0> at 0x7f713c063050> timeout=15, duration=75 at 0x7f7170698c90> task#=354 at 0x7f713c10ec90>, traceback: File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap self.__bootstrap_inner() File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File: "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run ret = func(*args, **kwargs) File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run self._execute_task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task task() File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__ self._callable() File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in __call__ self._func() File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in __call__ stats = hostapi.get_stats(self._cif, self._samples.stats()) File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats ret['haStats'] = _getHaInfo() File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo stats = instance.get_all_stats() File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats stats = broker.get_stats_from_storage() File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage result = self._proxy.get_stats() File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request verbose=self.__verbose File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request response = h.getresponse(buffering=True) File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse response.begin() File: "/usr/lib64/python2.7/httplib.py", line 444, in begin version, status, reason = self._read_status() File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status line = self.fp.readline(_MAXLINE + 1) File: "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize)

2 1