3:19 a.m.
On Tue, Dec 15, 2020 at 1:59 PM Alex K <rightkicktech(a)gmail.com> wrote:
On Tue, Dec 15, 2020 at 1:43 PM emesika <emesika(a)redhat.com> wrote:
> The problem is that the custom fencing configuration is not defined well
>
> Please follow [1] and retry
>
> [1]
> https://www.ovirt.org/develop/developer-guide/engine/custom-fencing.html
>
Yes, I followed that.
I cannot see what I am missing:
[root@manager ~]# engine-config -g CustomVdsFenceType
CustomVdsFenceType: fence_xvm version: general
should be only xvm
[root@manager ~]# engine-config -g CustomFenceAgentMapping
CustomFenceAgentMapping: fence_xvm=xvm version: general
Not needed , please keep empty
[root@manager ~]# engine-config -g CustomVdsFenceOptionMapping
CustomVdsFenceOptionMapping: fence_xvm: version: general
this one seems not OK , you should list here all options for the agent ,
please check the doc again
>
> On Tue, Dec 15, 2020 at 12:56 PM Alex K <rightkicktech(a)gmail.com> wrote:
>
>>
>>
>> On Tue, Dec 15, 2020 at 12:34 PM Martin Perina <mperina(a)redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Dec 15, 2020 at 11:18 AM Alex K <rightkicktech(a)gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Dec 15, 2020 at 11:59 AM Martin Perina
<mperina(a)redhat.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> could you please provide engine.log? And also vdsm.log from a host
>>>>> which was acting as a fence proxy?
>>>>>
>>>>
>>>> At proxy host (kvm1) I see the following vdsm.log:
>>>>
>>>> 2020-12-15 10:13:03,933+0000 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer]
>>>> RPC call Host.fenceNode failed (error 1) in 0.01 seconds (__init__:312)
>>>> 2020-12-15 10:13:04,376+0000 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer]
>>>> RPC call Host.fenceNode failed (error 1) in 0.01 seconds (__init__:312)
>>>>
>>>
>>> Isn't there stdout and stderr content of fence_xvm execution a few
>>> lines above, which should reveal the exact error? If not, then could you
>>> please turn on debug logging using below command:
>>>
>>> vdsm-client Host setLogLevel level=DEBUG
>>>
>>> This should be executed on the host which acts as a fence proxy (if you have
multiple hosts, then you would need to turn on debug on all, because the fence proxy is
selected randomly).
>>>
>>> Once we will have vdsm.log with fence_xvm execution details, then you can
change log level to INFO again by running:
>>>
>>> I had to set engine-config -s
CustomFenceAgentMapping="fence_xvm=xvm"
>> at engine, as it seems the host prepends fence_.
>> After that I got the following at the proxy host with DEBUG enabled:
>>
>> 2020-12-15 10:51:57,891+0000 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer]
>> Calling 'Host.fenceNode' in bridge with {u'username':
u'root', u'addr':
>> u'225.0.0.12', u'agent': u'xvm', u'options':
u'port=ovirt-node0',
>> u'action': u'status', u'password': '********',
u'port': u'0'} (__init__:329)
>> 2020-12-15 10:51:57,892+0000 DEBUG (jsonrpc/7) [root] /usr/bin/taskset
>> --cpu-list 0-3 /usr/sbin/fence_xvm (cwd None) (commands:198)
>> 2020-12-15 10:51:57,911+0000 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer]
>> RPC call Host.fenceNode failed (error 1) in 0.02 seconds (__init__:312)
>> 2020-12-15 10:51:58,339+0000 DEBUG (jsonrpc/5) [jsonrpc.JsonRpcServer]
>> Calling 'Host.fenceNode' in bridge with {u'username':
u'root', u'addr':
>> u'225.0.0.12', u'agent': u'xvm', u'options':
u'port=ovirt-node0',
>> u'action': u'status', u'password': '********',
u'port': u'0'} (__init__:329)
>> 2020-12-15 10:51:58,340+0000 DEBUG (jsonrpc/5) [root] /usr/bin/taskset
>> --cpu-list 0-3 /usr/sbin/fence_xvm (cwd None) (commands:198)
>> 2020-12-15 10:51:58,356+0000 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer]
>> RPC call Host.fenceNode failed (error 1) in 0.01 seconds (__init__:312
>>
>> while at engine at got:
>> 2020-12-15 10:51:57,873Z INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] EVENT_ID:
>> FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Executing power
>> management status on Host kvm0.lab.local using Proxy Host kvm1.lab.local
>> and Fence Agent xvm:225.0.0.12.
>> 2020-12-15 10:51:57,888Z INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>> task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] START,
>> FenceVdsVDSCommand(HostName = kvm1.lab.local,
>>
FenceVdsVDSCommandParameters:{hostId='91c81bbe-5933-4ed0-b9c5-2c8c277e44c7',
>> targetVdsId='b5e8fe3d-cbea-44cb-835a-f88d6d70c163',
action='STATUS',
>> agent='FenceAgent:{id='null', hostId='null',
order='1', type='xvm',
>> ip='225.0.0.12', port='0', user='root',
password='***',
>> encryptOptions='false', options='port=ovirt-node0'}',
policy='null'}), log
>> id: e6d3e8c
>> 2020-12-15 10:51:58,008Z WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] EVENT_ID:
>> VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host
>> kvm0.lab.local.Internal JSON-RPC error
>> 2020-12-15 10:51:58,008Z INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>> task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] FINISH, FenceVdsVDSCommand,
>> return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN',
>> message='Internal JSON-RPC error'}, log id: e6d3e8c
>> 2020-12-15 10:51:58,133Z WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] EVENT_ID:
>> FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Execution of power
>> management status on Host kvm0.lab.local using Proxy Host kvm1.lab.local
>> and Fence Agent xvm:225.0.0.12 failed.
>> 2020-12-15 10:51:58,134Z WARN
>> [org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-5)
>> [a4f30921-37a9-45c1-97e5-26152f844d72] Fence action failed using proxy host
>> 'kvm1.lab.local', trying another proxy
>> 2020-12-15 10:51:58,258Z ERROR
>> [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (default task-5)
>> [a4f30921-37a9-45c1-97e5-26152f844d72] Can not run fence action on host
>> 'kvm0.lab.local', no suitable proxy host was found.
>> 2020-12-15 10:51:58,258Z WARN
>> [org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-5)
>> [a4f30921-37a9-45c1-97e5-26152f844d72] Failed to find another proxy to
>> re-run failed fence action, retrying with the same proxy
'kvm1.lab.local'
>> 2020-12-15 10:51:58,334Z INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] EVENT_ID:
>> FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Executing power
>> management status on Host kvm0.lab.local using Proxy Host kvm1.lab.local
>> and Fence Agent xvm:225.0.0.12.
>> 2020-12-15 10:51:58,337Z INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>> task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] START,
>> FenceVdsVDSCommand(HostName = kvm1.lab.local,
>>
FenceVdsVDSCommandParameters:{hostId='91c81bbe-5933-4ed0-b9c5-2c8c277e44c7',
>> targetVdsId='b5e8fe3d-cbea-44cb-835a-f88d6d70c163',
action='STATUS',
>> agent='FenceAgent:{id='null', hostId='null',
order='1', type='xvm',
>> ip='225.0.0.12', port='0', user='root',
password='***',
>> encryptOptions='false', options='port=ovirt-node0'}',
policy='null'}), log
>> id: 557cbe7a
>> 2020-12-15 10:51:58,426Z WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] EVENT_ID:
>> VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host
>> kvm0.lab.local.Internal JSON-RPC error
>> 2020-12-15 10:51:58,427Z INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>> task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] FINISH, FenceVdsVDSCommand,
>> return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN',
>> message='Internal JSON-RPC error'}, log id: 557cbe7a
>> 2020-12-15 10:51:58,508Z WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [a4f30921-37a9-45c1-97e5-26152f844d72] EVENT_ID:
>> FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Execution of power
>> management status on Host kvm0.lab.local using Proxy Host kvm1.lab.local
>> and Fence Agent xvm:225.0.0.12 failed.
>>
>> I see that the proxy host uses twice the port option. Could that be the
>> reason?
>>
>>>
>>> vdsm-client Host setLogLevel level=INFO
>>>
>>> Thanks,
>>>
>>> Martin
>>>
>>> 2020-12-15 10:13:06,722+0000 INFO (jsonrpc/4) [api.host] FINISH
>>>> getStats return={'status': {'message': 'Done',
'code': 0}, 'info':
>>>> {'cpuStatistics': {'1': {'cpuUser':
'2.33', 'nodeIndex': 0, 'cpuSys':
>>>> '1.13', 'cpuIdle': '96.54'}, '0':
{'cpuUser': '1.66', 'nodeIndex': 0,
>>>> 'cpuSys': '0.47', 'cpuIdle': '97.87'},
'3': {'cpuUser': '0.73',
>>>> 'nodeIndex': 0, 'cpuSys': '0.60',
'cpuIdle': '98.67'}, '2': {'cpuUser':
>>>> '1.20', 'nodeIndex': 0, 'cpuSys': '0.40',
'cpuIdle': '98.40'}},
>>>> 'numaNodeMemFree': {'0': {'memPercent': 14,
'memFree': '8531'}},
>>>> 'memShared': 0, 'haScore': 3400, 'thpState':
'always',
>>>> 'ksmMergeAcrossNodes': True, 'vmCount': 0,
'memUsed': '8',
>>>> 'storageDomains':
{u'b4d25e5e-7806-464f-b2e1-4d4ab5a54dee': {'code': 0,
>>>> 'actual': True, 'version': 5, 'acquired': True,
'delay': '0.0027973',
>>>> 'lastCheck': '2.7', 'valid': True},
>>>> u'dc4d507b-954f-4da6-bcc3-b4f2633d0fa1': {'code': 0,
'actual': True,
>>>> 'version': 5, 'acquired': True, 'delay':
'0.00285824', 'lastCheck': '5.7',
>>>> 'valid': True}}, 'incomingVmMigrations': 0,
'network': {'ovirtmgmt':
>>>> {'rxErrors': '0', 'txErrors': '0',
'speed': '1000', 'rxDropped': '149',
>>>> 'name': 'ovirtmgmt', 'tx': '2980375',
'txDropped': '0', 'duplex':
>>>> 'unknown', 'sampleTime': 1608027186.703727, 'rx':
'27524740', 'state':
>>>> 'up'}, 'lo': {'rxErrors': '0',
'txErrors': '0', 'speed': '1000',
>>>> 'rxDropped': '0', 'name': 'lo',
'tx': '1085188922', 'txDropped': '0',
>>>> 'duplex': 'unknown', 'sampleTime':
1608027186.703727, 'rx': '1085188922',
>>>> 'state': 'up'}, 'ovs-system':
{'rxErrors': '0', 'txErrors': '0', 'speed':
>>>> '1000', 'rxDropped': '0', 'name':
'ovs-system', 'tx': '0', 'txDropped':
>>>> '0', 'duplex': 'unknown', 'sampleTime':
1608027186.703727, 'rx': '0',
>>>> 'state': 'down'}, ';vdsmdummy;':
{'rxErrors': '0', 'txErrors': '0',
>>>> 'speed': '1000', 'rxDropped': '0',
'name': ';vdsmdummy;', 'tx': '0',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027186.703727,
>>>> 'rx': '0', 'state': 'down'},
'br-int': {'rxErrors': '0', 'txErrors': '0',
>>>> 'speed': '1000', 'rxDropped': '0',
'name': 'br-int', 'tx': '0',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027186.703727,
>>>> 'rx': '0', 'state': 'down'},
'eth1': {'rxErrors': '0', 'txErrors': '0',
>>>> 'speed': '1000', 'rxDropped': '0',
'name': 'eth1', 'tx': '83685154',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027186.703727,
>>>> 'rx': '300648288', 'state': 'up'},
'eth0': {'rxErrors': '0', 'txErrors':
>>>> '0', 'speed': '1000', 'rxDropped':
'0', 'name': 'eth0', 'tx': '2980933',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027186.703727,
>>>> 'rx': '28271472', 'state': 'up'}},
'txDropped': '149', 'anonHugePages':
>>>> '182', 'ksmPages': 100, 'elapsedTime':
'5717.99', 'cpuLoad': '0.42',
>>>> 'cpuSys': '0.63', 'diskStats':
{'/var/log': {'free': '16444'},
>>>> '/var/run/vdsm/': {'free': '4909'},
'/tmp': {'free': '16444'}},
>>>> 'cpuUserVdsmd': '1.33', 'netConfigDirty':
'False', 'memCommitted': 0,
>>>> 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0,
'memAvailable': 9402,
>>>> 'bootTime': '1608021428', 'haStats':
{'active': True, 'configured': True,
>>>> 'score': 3400, 'localMaintenance': False,
'globalMaintenance': True},
>>>> 'momStatus': 'active', 'multipathHealth': {},
'rxDropped': '0',
>>>> 'outgoingVmMigrations': 0, 'swapTotal': 6015,
'swapFree': 6015,
>>>> 'hugepages': defaultdict(<type 'dict'>, {1048576:
{'resv_hugepages': 0,
>>>> 'free_hugepages': 0, 'nr_overcommit_hugepages': 0,
'surplus_hugepages': 0,
>>>> 'vm.free_hugepages': 0, 'nr_hugepages': 0,
'nr_hugepages_mempolicy': 0},
>>>> 2048: {'resv_hugepages': 0, 'free_hugepages': 0,
'nr_overcommit_hugepages':
>>>> 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0,
'nr_hugepages': 0,
>>>> 'nr_hugepages_mempolicy': 0}}), 'dateTime':
'2020-12-15T10:13:06 GMT',
>>>> 'cpuUser': '1.50', 'memFree': 9146,
'cpuIdle': '97.87', 'vmActive': 0,
>>>> 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}}
from=::1,55238 (api:54)
>>>> 2020-12-15 10:13:07,093+0000 INFO (jsonrpc/1) [api] FINISH getStats
>>>> error=Virtual machine does not exist: {'vmId':
>>>> u'0167fedb-7445-46bb-a39d-ea4471c86bf4'} (api:129)
>>>> 2020-12-15 10:13:07,094+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer]
>>>> RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
>>>> 2020-12-15 10:13:07,631+0000 INFO (jsonrpc/3) [api.host] FINISH
>>>> getStats return={'status': {'message': 'Done',
'code': 0}, 'info':
>>>> {'cpuStatistics': {'1': {'cpuUser':
'2.33', 'nodeIndex': 0, 'cpuSys':
>>>> '1.13', 'cpuIdle': '96.54'}, '0':
{'cpuUser': '1.66', 'nodeIndex': 0,
>>>> 'cpuSys': '0.47', 'cpuIdle': '97.87'},
'3': {'cpuUser': '0.73',
>>>> 'nodeIndex': 0, 'cpuSys': '0.60',
'cpuIdle': '98.67'}, '2': {'cpuUser':
>>>> '1.20', 'nodeIndex': 0, 'cpuSys': '0.40',
'cpuIdle': '98.40'}},
>>>> 'numaNodeMemFree': {'0': {'memPercent': 14,
'memFree': '8531'}},
>>>> 'memShared': 0, 'haScore': 3400, 'thpState':
'always',
>>>> 'ksmMergeAcrossNodes': True, 'vmCount': 0,
'memUsed': '8',
>>>> 'storageDomains':
{u'b4d25e5e-7806-464f-b2e1-4d4ab5a54dee': {'code': 0,
>>>> 'actual': True, 'version': 5, 'acquired': True,
'delay': '0.0027973',
>>>> 'lastCheck': '3.6', 'valid': True},
>>>> u'dc4d507b-954f-4da6-bcc3-b4f2633d0fa1': {'code': 0,
'actual': True,
>>>> 'version': 5, 'acquired': True, 'delay':
'0.00285824', 'lastCheck': '6.6',
>>>> 'valid': True}}, 'incomingVmMigrations': 0,
'network': {'ovirtmgmt':
>>>> {'rxErrors': '0', 'txErrors': '0',
'speed': '1000', 'rxDropped': '149',
>>>> 'name': 'ovirtmgmt', 'tx': '2985005',
'txDropped': '0', 'duplex':
>>>> 'unknown', 'sampleTime': 1608027187.616894, 'rx':
'27525820', 'state':
>>>> 'up'}, 'lo': {'rxErrors': '0',
'txErrors': '0', 'speed': '1000',
>>>> 'rxDropped': '0', 'name': 'lo',
'tx': '1085195824', 'txDropped': '0',
>>>> 'duplex': 'unknown', 'sampleTime':
1608027187.616894, 'rx': '1085195824',
>>>> 'state': 'up'}, 'ovs-system':
{'rxErrors': '0', 'txErrors': '0', 'speed':
>>>> '1000', 'rxDropped': '0', 'name':
'ovs-system', 'tx': '0', 'txDropped':
>>>> '0', 'duplex': 'unknown', 'sampleTime':
1608027187.616894, 'rx': '0',
>>>> 'state': 'down'}, ';vdsmdummy;':
{'rxErrors': '0', 'txErrors': '0',
>>>> 'speed': '1000', 'rxDropped': '0',
'name': ';vdsmdummy;', 'tx': '0',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027187.616894,
>>>> 'rx': '0', 'state': 'down'},
'br-int': {'rxErrors': '0', 'txErrors': '0',
>>>> 'speed': '1000', 'rxDropped': '0',
'name': 'br-int', 'tx': '0',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027187.616894,
>>>> 'rx': '0', 'state': 'down'},
'eth1': {'rxErrors': '0', 'txErrors': '0',
>>>> 'speed': '1000', 'rxDropped': '0',
'name': 'eth1', 'tx': '83689498',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027187.616894,
>>>> 'rx': '300653876', 'state': 'up'},
'eth0': {'rxErrors': '0', 'txErrors':
>>>> '0', 'speed': '1000', 'rxDropped':
'0', 'name': 'eth0', 'tx': '2985215',
>>>> 'txDropped': '0', 'duplex': 'unknown',
'sampleTime': 1608027187.616894,
>>>> 'rx': '28272664', 'state': 'up'}},
'txDropped': '149', 'anonHugePages':
>>>> '182', 'ksmPages': 100, 'elapsedTime':
'5718.91', 'cpuLoad': '0.42',
>>>> 'cpuSys': '0.63', 'diskStats':
{'/var/log': {'free': '16444'},
>>>> '/var/run/vdsm/': {'free': '4909'},
'/tmp': {'free': '16444'}},
>>>> 'cpuUserVdsmd': '1.33', 'netConfigDirty':
'False', 'memCommitted': 0,
>>>> 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0,
'memAvailable': 9402,
>>>> 'bootTime': '1608021428', 'haStats':
{'active': True, 'configured': True,
>>>> 'score': 3400, 'localMaintenance': False,
'globalMaintenance': True},
>>>> 'momStatus': 'active', 'multipathHealth': {},
'rxDropped': '0',
>>>> 'outgoingVmMigrations': 0, 'swapTotal': 6015,
'swapFree': 6015,
>>>> 'hugepages': defaultdict(<type 'dict'>, {1048576:
{'resv_hugepages': 0,
>>>> 'free_hugepages': 0, 'nr_overcommit_hugepages': 0,
'surplus_hugepages': 0,
>>>> 'vm.free_hugepages': 0, 'nr_hugepages': 0,
'nr_hugepages_mempolicy': 0},
>>>> 2048: {'resv_hugepages': 0, 'free_hugepages': 0,
'nr_overcommit_hugepages':
>>>> 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0,
'nr_hugepages': 0,
>>>> 'nr_hugepages_mempolicy': 0}}), 'dateTime':
'2020-12-15T10:13:07 GMT',
>>>> 'cpuUser': '1.50', 'memFree': 9146,
'cpuIdle': '97.87', 'vmActive': 0,
>>>> 'v2vJobs': {}, 'cpuSysVdsmd': '0.60'}}
from=::1,55238 (api:54)
>>>>
>>>> While at engine I have:
>>>> 2020-12-15 10:09:57,393Z ERROR
>>>> [org.ovirt.engine.core.utils.pm.VdsFenceOptions] (default task-13)
>>>> [fa61ae72-bc0c-4487-aeec-2b847877b6b5] Cannot find fence agent named
>>>> 'fence_xvm' in fence option mapping
>>>> 2020-12-15 10:09:57,519Z WARN
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>> (default task-13) [fa61ae72-bc0c-4487-aeec-2b847877b6b5] EVENT_ID:
>>>> VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for
Host
>>>> kvm0.lab.local.Internal JSON-RPC error
>>>> 2020-12-15 10:09:57,519Z INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>>>> task-13) [fa61ae72-bc0c-4487-aeec-2b847877b6b5] FINISH,
FenceVdsVDSCommand,
>>>> return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN',
>>>> message='Internal JSON-RPC error'}, log id: dc98f7c
>>>> 2020-12-15 10:09:57,596Z WARN
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>> (default task-13) [fa61ae72-bc0c-4487-aeec-2b847877b6b5] EVENT_ID:
>>>> FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Execution of power
>>>> management status on Host kvm0.lab.local using Proxy Host kvm1.lab.local
>>>> and Fence Agent fence_xvm:225.0.0.12 failed.
>>>> 2020-12-15 10:09:57,596Z WARN
>>>> [org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-13)
>>>> [fa61ae72-bc0c-4487-aeec-2b847877b6b5] Fence action failed using proxy
host
>>>> 'kvm1.lab.local', trying another proxy
>>>> 2020-12-15 10:09:57,694Z ERROR
>>>> [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (default task-13)
>>>> [fa61ae72-bc0c-4487-aeec-2b847877b6b5] Can not run fence action on host
>>>> 'kvm0.lab.local', no suitable proxy host was found.
>>>> 2020-12-15 10:09:57,695Z WARN
>>>> [org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-13)
>>>> [fa61ae72-bc0c-4487-aeec-2b847877b6b5] Failed to find another proxy to
>>>> re-run failed fence action, retrying with the same proxy
'kvm1.lab.local'
>>>> 2020-12-15 10:09:57,695Z ERROR
>>>> [org.ovirt.engine.core.utils.pm.VdsFenceOptions] (default task-13)
>>>> [fa61ae72-bc0c-4487-aeec-2b847877b6b5] Cannot find fence agent named
>>>> 'fence_xvm' in fence option mapping
>>>> 2020-12-15 10:09:57,815Z WARN
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>> (default task-13) [fa61ae72-bc0c-4487-aeec-2b847877b6b5] EVENT_ID:
>>>> VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for
Host
>>>> kvm0.lab.local.Internal JSON-RPC error
>>>> 2020-12-15 10:09:57,816Z INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>>>> task-13) [fa61ae72-bc0c-4487-aeec-2b847877b6b5] FINISH,
FenceVdsVDSCommand,
>>>> return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN',
>>>> message='Internal JSON-RPC error'}, log id: 4b58ec5e
>>>> 2020-12-15 10:09:57,895Z WARN
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>> (default task-13) [fa61ae72-bc0c-4487-aeec-2b847877b6b5] EVENT_ID:
>>>> FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Execution of power
>>>> management status on Host kvm0.lab.local using Proxy Host kvm1.lab.local
>>>> and Fence Agent fence_xvm:225.0.0.12 failed.
>>>>
>>>> At engine I had set the fence agent mapping as below (and have
>>>> restarted ovirt-engine service):
>>>>
>>>> engine-config -g CustomFenceAgentMapping
>>>> CustomFenceAgentMapping: fence_xvm=fence_xvm version: general
>>>>
>>>> Let me know if you need more logs.
>>>> I am running ovirt 4.3.10.
>>>>
>>>>
>>>>> Thanks,
>>>>> Martin
>>>>>
>>>>>
>>>>> On Tue, Dec 15, 2020 at 10:23 AM Alex K
<rightkicktech(a)gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Dec 15, 2020 at 11:07 AM Alex K
<rightkicktech(a)gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Dec 14, 2020 at 8:59 PM Strahil Nikolov <
>>>>>>> hunter86_bg(a)yahoo.com> wrote:
>>>>>>>
>>>>>>>> Fence_xvm requires a key is deployed on both the Host and
the VMs
>>>>>>>> in order to succeed. What is happening when you use the
cli on any of the
>>>>>>>> VMs ?
>>>>>>>> Also, the VMs require an open tcp port to receive the
necessary
>>>>>>>> output of each request.I
>>>>>>>
>>>>>>> I deployed keys at the physical host and virtual hosts, as
per
>>>>>>>
https://github.com/rightkick/Notes/blob/master/Ovirt-fence_xmv.md
>>>>>>> I can get the VM status from the virtual hosts:
>>>>>>>
>>>>>>> [root@kvm1 cluster]# fence_xvm -a 225.0.0.12 -k
>>>>>>> /etc/cluster/fence_xvm.key -H ovirt-node0 -o status
>>>>>>> Status: ON
>>>>>>> You have new mail in /var/spool/mail/root
>>>>>>> [root@kvm1 cluster]# fence_xvm -a 225.0.0.12 -k
>>>>>>> /etc/cluster/fence_xvm.key -H ovirt-node1 -o status
>>>>>>> Status: ON
>>>>>>>
>>>>>>> kvm0 and kvm1 are the hostnames of each virtual host, while
>>>>>>> ovirt-node0 and ovirt-node1 are the domain names of the same
virtual hosts
>>>>>>> as defined at virsh.
>>>>>>>
>>>>>>
>>>>>> I am passing also the port/domain option at GUI, but from logs
it
>>>>>> seems it is being ignored as it is not being logged from engine.
>>>>>>
>>>>>> [image: image.png]
>>>>>> tried also domain=ovirt-node0 with same results.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Strahil Nikolov
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> В понеделник, 14 декември 2020 г., 10:57:11 Гринуич+2,
Alex K <
>>>>>>>> rightkicktech(a)gmail.com> написа:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi friends,
>>>>>>>>
>>>>>>>> I was wondering what is needed to setup fence_xvm in
order to use
>>>>>>>> for power management in virtual nested environments for
testing purposes.
>>>>>>>>
>>>>>>>> I have followed the following steps:
>>>>>>>>
https://github.com/rightkick/Notes/blob/master/Ovirt-fence_xmv.md
>>>>>>>>
>>>>>>>> I tried also engine-config -s
>>>>>>>> CustomFenceAgentMapping="fence_xvm=_fence_xvm"
>>>>>>>> From command line all seems fine and I can get the status
of the
>>>>>>>> host VMs, but I was not able to find what is needed to
set this up at
>>>>>>>> engine UI:
>>>>>>>>
>>>>>>>>
>>>>>>>> At username and pass I just filled dummy values as they
should not
>>>>>>>> be needed for fence_xvm.
>>>>>>>> I always get an error at GUI while engine logs give:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2020-12-14 08:53:48,343Z WARN
>>>>>>>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>>>>> (default task-4) [07c1d540-6d8d-419c-affb-181495d75759]
EVENT_ID:
>>>>>>>> VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test
failed for Host
>>>>>>>> kvm0.lab.local.Internal JSON-RPC error
>>>>>>>> 2020-12-14 08:53:48,343Z INFO
>>>>>>>>
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>>>>>>>> task-4) [07c1d540-6d8d-419c-affb-181495d75759] FINISH,
FenceVdsVDSCommand,
>>>>>>>> return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN',
>>>>>>>> message='Internal JSON-RPC error'}, log id:
2437b13c
>>>>>>>> 2020-12-14 08:53:48,400Z WARN
>>>>>>>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>>>>> (default task-4) [07c1d540-6d8d-419c-affb-181495d75759]
EVENT_ID:
>>>>>>>> FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021),
Execution of power
>>>>>>>> management status on Host kvm0.lab.local using Proxy Host
kvm1.lab.local
>>>>>>>> and Fence Agent fence_xvm:225.0.0.12 failed.
>>>>>>>> 2020-12-14 08:53:48,400Z WARN
>>>>>>>> [org.ovirt.engine.core.bll.pm.FenceAgentExecutor]
(default task-4)
>>>>>>>> [07c1d540-6d8d-419c-affb-181495d75759] Fence action
failed using proxy host
>>>>>>>> 'kvm1.lab.local', trying another proxy
>>>>>>>> 2020-12-14 08:53:48,485Z ERROR
[org.ovirt.engine.core.bll.pm.FenceProxyLocator]
>>>>>>>> (default task-4) [07c1d540-6d8d-419c-affb-181495d75759]
Can not run fence
>>>>>>>> action on host 'kvm0.lab.local', no suitable
proxy host was found.
>>>>>>>> 2020-12-14 08:53:48,486Z WARN
>>>>>>>> [org.ovirt.engine.core.bll.pm.FenceAgentExecutor]
(default task-4)
>>>>>>>> [07c1d540-6d8d-419c-affb-181495d75759] Failed to find
another proxy to
>>>>>>>> re-run failed fence action, retrying with the same proxy
'kvm1.lab.local'
>>>>>>>> 2020-12-14 08:53:48,582Z WARN
>>>>>>>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>>>>> (default task-4) [07c1d540-6d8d-419c-affb-181495d75759]
EVENT_ID:
>>>>>>>> VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test
failed for Host
>>>>>>>> kvm0.lab.local.Internal JSON-RPC error
>>>>>>>> 2020-12-14 08:53:48,582Z INFO
>>>>>>>>
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
>>>>>>>> task-4) [07c1d540-6d8d-419c-affb-181495d75759] FINISH,
FenceVdsVDSCommand,
>>>>>>>> return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN',
>>>>>>>> message='Internal JSON-RPC error'}, log id:
8607bc9
>>>>>>>> 2020-12-14 08:53:48,637Z WARN
>>>>>>>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>>>>>>> (default task-4) [07c1d540-6d8d-419c-affb-181495d75759]
EVENT_ID:
>>>>>>>> FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021),
Execution of power
>>>>>>>> management status on Host kvm0.lab.local using Proxy Host
kvm1.lab.local
>>>>>>>> and Fence Agent fence_xvm:225.0.0.12 failed.
>>>>>>>>
>>>>>>>>
>>>>>>>> Any idea?
>>>>>>>>
>>>>>>>> Thanx,
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list -- users(a)ovirt.org
>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>>>>>> Privacy Statement:
https://www.ovirt.org/privacy-policy.html
>>>>>>>> oVirt Code of Conduct:
>>>>>>>>
https://www.ovirt.org/community/about/community-guidelines/
>>>>>>>> List Archives:
>>>>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/B7IHC4MYY5L...
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>> Users mailing list -- users(a)ovirt.org
>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>>>>> oVirt Code of Conduct:
>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>> List Archives:
>>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MV3RI22LE4C...
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Martin Perina
>>>>> Manager, Software Engineering
>>>>> Red Hat Czech s.r.o.
>>>>>
>>>>
>>>
>>> --
>>> Martin Perina
>>> Manager, Software Engineering
>>> Red Hat Czech s.r.o.
>>>
>>
