Info on fence_rhevm against oVirt 4.1.1
Hello, I'm trying to use fence_rhevm in a CentOS 6.8 guest that is part of a virtual rhcs cluster My sw version for fence_agents inside guest is fence-agents-4.0.15-12.el6.x86_64 and I notice that for this particular agent nothing changes also using the latest available package fence-agents-4.0.15-13.el6.x86_64.rpm apart [root@p2vnorasvi1 ~]# diff fence_rhevm /usr/sbin/fence_rhevm 13c13 < BUILD_DATE="(built Wed Mar 22 04:24:11 UTC 2017)" ---
BUILD_DATE="(built Tue May 10 22:28:47 UTC 2016)" [root@p2vnorasvi1 ~]#
The VM name in oVirt 4.1.1 is p2vorasvi1 Running this command against the engine I get [root@p2vnorasvi1 network-scripts]# fence_rhevm -a 10.4.192.43 -l "admin@internal" -p "mypassword" -z --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /api/vms/ was not found on this server.</p> </body></html> Failed: Unable to obtain correct plug status or plug is not available Actually I get the same error even if I put a wrong password.... What am I missing...? Do I have to specify DC/cluster if I have more than one, or other parameters? Thanks, Gianluca
On Thu, Apr 27, 2017 at 4:43 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, I'm trying to use fence_rhevm in a CentOS 6.8 guest that is part of a virtual rhcs cluster
My sw version for fence_agents inside guest is fence-agents-4.0.15-12.el6.x86_64 and I notice that for this particular agent nothing changes also using the latest available package fence-agents-4.0.15-13.el6.x86_64.rpm apart
[root@p2vnorasvi1 ~]# diff fence_rhevm /usr/sbin/fence_rhevm 13c13 < BUILD_DATE="(built Wed Mar 22 04:24:11 UTC 2017)" ---
BUILD_DATE="(built Tue May 10 22:28:47 UTC 2016)" [root@p2vnorasvi1 ~]#
The VM name in oVirt 4.1.1 is p2vorasvi1
Running this command against the engine I get
[root@p2vnorasvi1 network-scripts]# fence_rhevm -a 10.4.192.43 -l "admin@internal" -p "mypassword" -z --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /api/vms/ was not found on this server.</p> </body></html>
Failed: Unable to obtain correct plug status or plug is not available
Actually I get the same error even if I put a wrong password....
What am I missing...? Do I have to specify DC/cluster if I have more than one, or other parameters?
Thanks, Gianluca
If I change this in fence_rhevm [root@p2vnorasvi1 sbin]# diff fence_rhevm fence_rhevm.orig 84c84 < url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/ovirt-engine/api/" + command ---
url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/api/" + command
I now get 401 unauthorized.... [root@p2vnorasvi1 sbin]# fence_rhevm -a 10.4.192.43 -z -l "admin@internal" -p "mypassword" --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1 <html><head><title>Error</title></head><body>Unauthorized</body></html> Failed: Unable to obtain correct plug status or plug is not available [root@p2vnorasvi1 sbin]# and in engine ssl_access.log 127.0.0.1 - - [27/Apr/2017:16:51:55 +0200] "POST /ovirt-engine/sso/oauth/token HTTP/1.1" 200 153 10.4.168.91 - - [27/Apr/2017:16:51:55 +0200] "GET /ovirt-engine/api/vms/?search=name%3Dp2vorasvi2 HTTP/1.1" 401 71
On Thu, Apr 27, 2017 at 4:58 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Apr 27, 2017 at 4:43 PM, Gianluca Cecchi < gianluca.cecchi@gmail.com> wrote:
Hello, I'm trying to use fence_rhevm in a CentOS 6.8 guest that is part of a virtual rhcs cluster
My sw version for fence_agents inside guest is fence-agents-4.0.15-12.el6.x86_64 and I notice that for this particular agent nothing changes also using the latest available package fence-agents-4.0.15-13.el6.x86_64.rpm apart
[root@p2vnorasvi1 ~]# diff fence_rhevm /usr/sbin/fence_rhevm 13c13 < BUILD_DATE="(built Wed Mar 22 04:24:11 UTC 2017)" ---
BUILD_DATE="(built Tue May 10 22:28:47 UTC 2016)" [root@p2vnorasvi1 ~]#
The VM name in oVirt 4.1.1 is p2vorasvi1
Running this command against the engine I get
[root@p2vnorasvi1 network-scripts]# fence_rhevm -a 10.4.192.43 -l "admin@internal" -p "mypassword" -z --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /api/vms/ was not found on this server.</p> </body></html>
Failed: Unable to obtain correct plug status or plug is not available
Actually I get the same error even if I put a wrong password....
What am I missing...? Do I have to specify DC/cluster if I have more than one, or other parameters?
Thanks, Gianluca
If I change this in fence_rhevm
[root@p2vnorasvi1 sbin]# diff fence_rhevm fence_rhevm.orig 84c84 < url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/ovirt-engine/api/" + command ---
url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/api/" + command
I now get 401 unauthorized....
[root@p2vnorasvi1 sbin]# fence_rhevm -a 10.4.192.43 -z -l "admin@internal" -p "mypassword" --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1
<html><head><title>Error</title></head><body>Unauthorized</body></html>
Failed: Unable to obtain correct plug status or plug is not available
[root@p2vnorasvi1 sbin]#
and in engine ssl_access.log
127.0.0.1 - - [27/Apr/2017:16:51:55 +0200] "POST /ovirt-engine/sso/oauth/token HTTP/1.1" 200 153 10.4.168.91 - - [27/Apr/2017:16:51:55 +0200] "GET /ovirt-engine/api/vms/?search=name%3Dp2vorasvi2 HTTP/1.1" 401 71
Tried also using v3 in url, this way: [root@p2vnorasvi1 sbin]# diff fence_rhevm fence_rhevm.orig 84c84 < url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/ovirt-engine/api/v3/" + command ---
url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/api/" + command [root@p2vnorasvi1 sbin]#
[root@p2vnorasvi1 sbin]# fence_rhevm -a 10.4.192.43 -z -l "admin@internal" -p "mypassword" --shell-timeout=20 --login-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1 <html><head><title>Error</title></head><body>Unauthorized</body></html> Failed: Unable to obtain correct plug status or plug is not available [root@p2vnorasvi1 sbin]#
On 04/27/2017 05:35 PM, Gianluca Cecchi wrote:
On Thu, Apr 27, 2017 at 4:58 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:
On Thu, Apr 27, 2017 at 4:43 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:
Hello, I'm trying to use fence_rhevm in a CentOS 6.8 guest that is part of a virtual rhcs cluster
My sw version for fence_agents inside guest is fence-agents-4.0.15-12.el6.x86_64 and I notice that for this particular agent nothing changes also using the latest available package fence-agents-4.0.15-13.el6.x86_64.rpm apart
[root@p2vnorasvi1 ~]# diff fence_rhevm /usr/sbin/fence_rhevm 13c13 < BUILD_DATE="(built Wed Mar 22 04:24:11 UTC 2017)" --- > BUILD_DATE="(built Tue May 10 22:28:47 UTC 2016)" [root@p2vnorasvi1 ~]#
The VM name in oVirt 4.1.1 is p2vorasvi1
Running this command against the engine I get
[root@p2vnorasvi1 network-scripts]# fence_rhevm -a 10.4.192.43 -l "admin@internal" -p "mypassword" -z --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /api/vms/ was not found on this server.</p> </body></html>
Failed: Unable to obtain correct plug status or plug is not available
Actually I get the same error even if I put a wrong password....
What am I missing...? Do I have to specify DC/cluster if I have more than one, or other parameters?
Thanks, Gianluca
If I change this in fence_rhevm
[root@p2vnorasvi1 sbin]# diff fence_rhevm fence_rhevm.orig 84c84 < url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/ovirt-engine/api/" + command --- > url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/api/" + command
I now get 401 unauthorized....
[root@p2vnorasvi1 sbin]# fence_rhevm -a 10.4.192.43 -z -l "admin@internal" -p "mypassword" --shell-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1
<html><head><title>Error</title></head><body>Unauthorized</body></html>
Failed: Unable to obtain correct plug status or plug is not available
[root@p2vnorasvi1 sbin]#
and in engine ssl_access.log
127.0.0.1 - - [27/Apr/2017:16:51:55 +0200] "POST /ovirt-engine/sso/oauth/token HTTP/1.1" 200 153 10.4.168.91 - - [27/Apr/2017:16:51:55 +0200] "GET /ovirt-engine/api/vms/?search=name%3Dp2vorasvi2 HTTP/1.1" 401 71
Tried also using v3 in url, this way:
[root@p2vnorasvi1 sbin]# diff fence_rhevm fence_rhevm.orig 84c84 < url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/ovirt-engine/api/v3/" + command ---
url += "//" + opt["--ip"] + ":" + str(opt["--ipport"]) + "/api/" + command [root@p2vnorasvi1 sbin]#
[root@p2vnorasvi1 sbin]# fence_rhevm -a 10.4.192.43 -z -l "admin@internal" -p "mypassword" --shell-timeout=20 --login-timeout=20 --power-wait=10 -v -o status -n p2vorasvi1 vms/?search=name%3Dp2vorasvi1
<html><head><title>Error</title></head><body>Unauthorized</body></html>
Failed: Unable to obtain correct plug status or plug is not available
[root@p2vnorasvi1 sbin]#
That is a known issue: fence_rhevm can only work as RHEV admin user not a regular user (that requires "Filter: true http header) https://bugzilla.redhat.com/1287059 That was fixed in fence-agents-4.0.11-47.el7, but I guess it wasn't backported to CentOS 6. I'd suggest that you open a bug for this component in the Red Hat Enterprise Linux bug tracker, requesting that the fix be back-ported. Meanwhile, if you are in a hurry, you can take the CentOS 7 fence_rhev script, which should work. You will most likely also need to add --ssl-indecure to the command line of the agent, because you will most likely be using the default self signed certificate authority used by the engine. Note that the latest version of this script uses the 'Filter: true' header to drop privileges. That means that even when using 'admin@internal' you have to make sure that 'admin@internal' has permissions for the VM that you want to fence, otherwise it will not be able to find/fence it.
On Thu, Apr 27, 2017 at 6:32 PM, Juan Hernández <jhernand@redhat.com> wrote:
That is a known issue:
fence_rhevm can only work as RHEV admin user not a regular user (that requires "Filter: true http header) https://bugzilla.redhat.com/1287059
That was fixed in fence-agents-4.0.11-47.el7, but I guess it wasn't backported to CentOS 6.
I'd suggest that you open a bug for this component in the Red Hat Enterprise Linux bug tracker, requesting that the fix be back-ported.
Meanwhile, if you are in a hurry, you can take the CentOS 7 fence_rhev script, which should work.
You will most likely also need to add --ssl-indecure to the command line of the agent, because you will most likely be using the default self signed certificate authority used by the engine.
Note that the latest version of this script uses the 'Filter: true' header to drop privileges. That means that even when using 'admin@internal' you have to make sure that 'admin@internal' has permissions for the VM that you want to fence, otherwise it will not be able to find/fence it.
Thanks for the feedback Juan. I confirm that using fence_rhevm from latest CentOS 7 version it worked. These were the lines in my cluster.conf <clusternode name="p2viclnorasvi1" nodeid="1" votes="1"> <fence> <method name="1"> <device name="ovirt_fencedelay" port="p2vorasvi1"/> </method> </fence> </clusternode> <clusternode name="p2viclnorasvi2" nodeid="2" votes="1"> <fence> <method name="1"> <device name="ovirt_fence" port="p2vorasvi2"/> </method> </fence> </clusternode> </clusternodes> <quorumd label="p2vcluorasvi" votes="1"> <heuristic interval="2" program="ping -c1 -w1 172.16.10.231" score="1" tko="200"/> </quorumd> <fencedevices> <fencedevice agent="fence_rhevm" delay="30" ipaddr="10.4.192.43" login="g.cecchi@internal" passwd_script="/usr/local/bin/pwd_dracnode01.sh" name="ovirt_fencedelay" ssl="on" ssl_insecure="on" shell_timeout="20" power_wait="10"/> <fencedevice agent="fence_rhevm" ipaddr="10.4.192.43" login="g.cecchi@internal" passwd_script="/usr/local/bin/pwd_dracnode02.sh" name="ovirt_fence" ssl="on" ssl_insecure="on" shell_timeout="20" power_wait="10"/> </fencedevices> Using admin@internal didn't work even if I set the permissions at vm level too... It worked with my username (g.cecchi) that has SuperUser system privilege and also at VM level. Is it yet necessary to have a user with SuperUser privilege at system level? Tomorrow (today... ;-) I'm going to open a bugzilla to backport the feature. Thanks again, Gianluca
On 04/28/2017 01:54 AM, Gianluca Cecchi wrote:
On Thu, Apr 27, 2017 at 6:32 PM, Juan Hernández <jhernand@redhat.com <mailto:jhernand@redhat.com>> wrote:
That is a known issue:
fence_rhevm can only work as RHEV admin user not a regular user (that requires "Filter: true http header) https://bugzilla.redhat.com/1287059 <https://bugzilla.redhat.com/1287059>
That was fixed in fence-agents-4.0.11-47.el7, but I guess it wasn't backported to CentOS 6.
I'd suggest that you open a bug for this component in the Red Hat Enterprise Linux bug tracker, requesting that the fix be back-ported.
Meanwhile, if you are in a hurry, you can take the CentOS 7 fence_rhev script, which should work.
You will most likely also need to add --ssl-indecure to the command line of the agent, because you will most likely be using the default self signed certificate authority used by the engine.
Note that the latest version of this script uses the 'Filter: true' header to drop privileges. That means that even when using 'admin@internal' you have to make sure that 'admin@internal' has permissions for the VM that you want to fence, otherwise it will not be able to find/fence it.
Thanks for the feedback Juan. I confirm that using fence_rhevm from latest CentOS 7 version it worked. These were the lines in my cluster.conf
<clusternode name="p2viclnorasvi1" nodeid="1" votes="1"> <fence> <method name="1"> <device name="ovirt_fencedelay" port="p2vorasvi1"/> </method> </fence> </clusternode> <clusternode name="p2viclnorasvi2" nodeid="2" votes="1"> <fence> <method name="1"> <device name="ovirt_fence" port="p2vorasvi2"/> </method> </fence> </clusternode> </clusternodes> <quorumd label="p2vcluorasvi" votes="1"> <heuristic interval="2" program="ping -c1 -w1 172.16.10.231" score="1" tko="200"/> </quorumd> <fencedevices> <fencedevice agent="fence_rhevm" delay="30" ipaddr="10.4.192.43" login="g.cecchi@internal" passwd_script="/usr/local/bin/pwd_dracnode01.sh" name="ovirt_fencedelay" ssl="on" ssl_insecure="on" shell_timeout="20" power_wait="10"/> <fencedevice agent="fence_rhevm" ipaddr="10.4.192.43" login="g.cecchi@internal" passwd_script="/usr/local/bin/pwd_dracnode02.sh" name="ovirt_fence" ssl="on" ssl_insecure="on" shell_timeout="20" power_wait="10"/> </fencedevices>
Using admin@internal didn't work even if I set the permissions at vm level too...
It should work adding 'UserRole' to 'admin@internal'. The issue is that the fence agent uses the 'Filter: true' header, thus it drops its super-user privileges to do the query, and won't get the VM unless it has explicitly granted permissions. To check it you can do the following, for example: ---8<--- #!/bin/sh -ex url="https://yourengine/ovirt-engine/api" user="admin@internal" password="..." curl \ --verbose \ --cacert "/etc/pki/ovirt-engine/ca.pem" \ --user "${user}:${password}" \ --request GET \ --header "Version: 3" \ --header "Filter: true" \ "${url}/vms?search=name%3Dmyvm" --->8--- That should return the details of the VM, or nothing if the user doesn't have permission to see that VM.
It worked with my username (g.cecchi) that has SuperUser system privilege and also at VM level.
Is it yet necessary to have a user with SuperUser privilege at system level?
No, it shouldn't be necessary. Actually, as you are using the 'internal' domain, it is easy to add a new dummy user without SuperUser privileges. You can give that user permissions (with 'UserRole') only for the VMs that are nodes of the cluster. That should be enough.
Tomorrow (today... ;-) I'm going to open a bugzilla to backport the feature.
Thanks again, Gianluca
Using admin@internal didn't work even if I set the permissions at vm level too...
It worked with my username (g.cecchi) that has SuperUser system privilege and also at VM level.
Is it yet necessary to have a user with SuperUser privilege at system level?
Tomorrow (today... ;-) I'm going to open a bugzilla to backport the feature.
Thanks again, Gianluca
In the mean time I have opened this: https://bugzilla.redhat.com/show_bug.cgi?id=1446474 Feel free to add any comment. I would test your suggestions. It seems I already tried them with a "fenceuser" user, but they didn't work, but it was late in the night and possibly I missed something... ;-) It has been very nice in less than a day to make a "poor man" P2V of a physical 2-nodes RHCS running cluster into oVirt and be able to reproduce and solve a problem we had during maintenance windows with clvmd not starting due to timeout issues. And also with both production and intracluster networks of the virtual cluster based on OVN, without having to spend time waiting for network support, due to time constraints we had, as the next maintenance window will be today... thanks to all Gianluca
participants (2)
-
Gianluca Cecchi -
Juan Hernández