--------------DDA22B530AEEAC5C11E37F1E Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit I'm adding 3 hardware nodes to our cluster. All 3 same type of server and software, HP DL360 G8, centos 7. One fails the fence agent test. The one I'm having problems with has a newer version of ilo firmware, not sure if related. Troublemaker: 2.54 Others: 2.53 ovirt-engine-4.1.0.4-1.el7.centos.noarch vdsm-4.19.4-1.el7.centos.x86_64 2017-07-26 15:14:30,215-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing power management status on Host ovirt6.j2noc.com using Proxy Host ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89. 2017-07-26 15:14:30,216-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START, FenceVdsVDSCommand(HostName = ovirt1.j2noc.com, FenceVdsVDSCommandParameters:{runAsync='true', hostId='23d2c0ab-5dd1-43af-9db3-2a426a539faf', targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS', agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4', ip='10.144.254.89', port='null', user='Administrator', password='***', encryptOptions='false', options='power_wait=4'}', policy='null'}), log id: 1498b3c4 *2017-07-26 15:14:30,414-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct plug status or plug is not available, , ]'}, log id: 1498b3c4* 2017-07-26 15:14:30,420-07 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of power management status on Host ovirt6.j2noc.com using Proxy Host ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89 failed. 2017-07-26 15:14:30,420-07 WARN [org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] Fence action failed using proxy host '10.144.110.99', trying another proxy 2017-07-26 15:14:30,740-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing power management status on Host ovirt6.j2noc.com using Proxy Host ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89. 2017-07-26 15:14:30,741-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START, FenceVdsVDSCommand(HostName = ovirt2.j2noc.com, FenceVdsVDSCommandParameters:{runAsync='true', hostId='91d8fa70-fd24-4530-90f7-982ff068230b', targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS', agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4', ip='10.144.254.89', port='null', user='Administrator', password='***', encryptOptions='false', options='power_wait=4'}', policy='null'}), log id: 67d837da 2017-07-26 15:14:30,898-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct plug status or plug is not available, , ]'}, log id: 67d837da 2017-07-26 15:14:30,903-07 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of power management status on Host ovirt6.j2noc.com using Proxy Host ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89 failed. I'm not sure the right syntax for fence_ipmilan since even a "good" host fails: [root@ovirt4 prod vdsm]# fence_ipmilan -a 10.144.254.87 -P -l Administrator -p *** -o status -v chassis power status Executing: /usr/bin/ipmitool -I lanplus -H 10.144.254.87 -U Administrator -P [set] -p 623 -L ADMINISTRATOR chassis power status 1 Error: Unable to establish IPMI v2 / RMCP+ session Failed: Unable to obtain correct plug status or plug is not available Any ideas on what the issue is? --------------DDA22B530AEEAC5C11E37F1E Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> I'm adding 3 hardware nodes to our cluster. All 3 same type of server and software, HP DL360 G8, centos 7.<br> One fails the fence agent test.<br> The one I'm having problems with has a newer version of ilo firmware, not sure if related.<br> Troublemaker: 2.54<br> Others: 2.53<br> <br> ovirt-engine-4.1.0.4-1.el7.centos.noarch<br> vdsm-4.19.4-1.el7.centos.x86_64<br> <br> <br> 2017-07-26 15:14:30,215-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing power management status on Host ovirt6.j2noc.com using Proxy Host ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89.<br> 2017-07-26 15:14:30,216-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START, FenceVdsVDSCommand(HostName = ovirt1.j2noc.com, FenceVdsVDSCommandParameters:{runAsync='true', hostId='23d2c0ab-5dd1-43af-9db3-2a426a539faf', targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS', agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4', ip='10.144.254.89', port='null', user='Administrator', password='***', encryptOptions='false', options='power_wait=4'}', policy='null'}), log id: 1498b3c4<br> <b>2017-07-26 15:14:30,414-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct plug status or plug is not available, , ]'}, log id: 1498b3c4</b><br> 2017-07-26 15:14:30,420-07 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of power management status on Host ovirt6.j2noc.com using Proxy Host ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89 failed.<br> 2017-07-26 15:14:30,420-07 WARN [org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] Fence action failed using proxy host '10.144.110.99', trying another proxy<br> 2017-07-26 15:14:30,740-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing power management status on Host ovirt6.j2noc.com using Proxy Host ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89.<br> 2017-07-26 15:14:30,741-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START, FenceVdsVDSCommand(HostName = ovirt2.j2noc.com, FenceVdsVDSCommandParameters:{runAsync='true', hostId='91d8fa70-fd24-4530-90f7-982ff068230b', targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS', agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4', ip='10.144.254.89', port='null', user='Administrator', password='***', encryptOptions='false', options='power_wait=4'}', policy='null'}), log id: 67d837da<br> 2017-07-26 15:14:30,898-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR', powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct plug status or plug is not available, , ]'}, log id: 67d837da<br> 2017-07-26 15:14:30,903-07 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Execution of power management status on Host ovirt6.j2noc.com using Proxy Host ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89 failed.<br> <br> <br> <br> I'm not sure the right syntax for fence_ipmilan since even a "good" host fails:<br> <br> <br> [root@ovirt4 prod vdsm]# fence_ipmilan -a 10.144.254.87 -P -l Administrator -p *** -o status -v chassis power status <br> Executing: /usr/bin/ipmitool -I lanplus -H 10.144.254.87 -U Administrator -P [set] -p 623 -L ADMINISTRATOR chassis power status<br> <br> 1 Error: Unable to establish IPMI v2 / RMCP+ session<br> <br> <br> Failed: Unable to obtain correct plug status or plug is not available<br> <br> <br> <br> <br> Any ideas on what the issue is?<br> <br> </body> </html> --------------DDA22B530AEEAC5C11E37F1E--