fence issue adding host
by Bill James
--------------DDA22B530AEEAC5C11E37F1E
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
I'm adding 3 hardware nodes to our cluster. All 3 same type of server
and software, HP DL360 G8, centos 7.
One fails the fence agent test.
The one I'm having problems with has a newer version of ilo firmware,
not sure if related.
Troublemaker: 2.54
Others: 2.53
ovirt-engine-4.1.0.4-1.el7.centos.noarch
vdsm-4.19.4-1.el7.centos.x86_64
2017-07-26 15:14:30,215-07 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID:
null, Call Stack: null, Custom Event ID: -1, Message: Executing power
management status on Host ovirt6.j2noc.com using Proxy Host
ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89.
2017-07-26 15:14:30,216-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START,
FenceVdsVDSCommand(HostName = ovirt1.j2noc.com,
FenceVdsVDSCommandParameters:{runAsync='true',
hostId='23d2c0ab-5dd1-43af-9db3-2a426a539faf',
targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS',
agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4',
ip='10.144.254.89', port='null', user='Administrator', password='***',
encryptOptions='false', options='power_wait=4'}', policy='null'}), log
id: 1498b3c4
*2017-07-26 15:14:30,414-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH,
FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct plug
status or plug is not available, , ]'}, log id: 1498b3c4*
2017-07-26 15:14:30,420-07 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID:
null, Call Stack: null, Custom Event ID: -1, Message: Execution of power
management status on Host ovirt6.j2noc.com using Proxy Host
ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89 failed.
2017-07-26 15:14:30,420-07 WARN
[org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-52)
[4d72d360-2b92-43f6-b2df-e80ee305c622] Fence action failed using proxy
host '10.144.110.99', trying another proxy
2017-07-26 15:14:30,740-07 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID:
null, Call Stack: null, Custom Event ID: -1, Message: Executing power
management status on Host ovirt6.j2noc.com using Proxy Host
ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89.
2017-07-26 15:14:30,741-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START,
FenceVdsVDSCommand(HostName = ovirt2.j2noc.com,
FenceVdsVDSCommandParameters:{runAsync='true',
hostId='91d8fa70-fd24-4530-90f7-982ff068230b',
targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS',
agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4',
ip='10.144.254.89', port='null', user='Administrator', password='***',
encryptOptions='false', options='power_wait=4'}', policy='null'}), log
id: 67d837da
2017-07-26 15:14:30,898-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (default
task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH,
FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct plug
status or plug is not available, , ]'}, log id: 67d837da
2017-07-26 15:14:30,903-07 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID:
null, Call Stack: null, Custom Event ID: -1, Message: Execution of power
management status on Host ovirt6.j2noc.com using Proxy Host
ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89 failed.
I'm not sure the right syntax for fence_ipmilan since even a "good" host
fails:
[root@ovirt4 prod vdsm]# fence_ipmilan -a 10.144.254.87 -P -l
Administrator -p *** -o status -v chassis power status
Executing: /usr/bin/ipmitool -I lanplus -H 10.144.254.87 -U
Administrator -P [set] -p 623 -L ADMINISTRATOR chassis power status
1 Error: Unable to establish IPMI v2 / RMCP+ session
Failed: Unable to obtain correct plug status or plug is not available
Any ideas on what the issue is?
--------------DDA22B530AEEAC5C11E37F1E
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I'm adding 3 hardware nodes to our cluster. All 3 same type of
server and software, HP DL360 G8, centos 7.<br>
One fails the fence agent test.<br>
The one I'm having problems with has a newer version of ilo
firmware, not sure if related.<br>
Troublemaker: 2.54<br>
Others: 2.53<br>
<br>
ovirt-engine-4.1.0.4-1.el7.centos.noarch<br>
vdsm-4.19.4-1.el7.centos.x86_64<br>
<br>
<br>
2017-07-26 15:14:30,215-07 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation
ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing
power management status on Host ovirt6.j2noc.com using Proxy Host
ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89.<br>
2017-07-26 15:14:30,216-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START,
FenceVdsVDSCommand(HostName = ovirt1.j2noc.com,
FenceVdsVDSCommandParameters:{runAsync='true',
hostId='23d2c0ab-5dd1-43af-9db3-2a426a539faf',
targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS',
agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4',
ip='10.144.254.89', port='null', user='Administrator',
password='***', encryptOptions='false', options='power_wait=4'}',
policy='null'}), log id: 1498b3c4<br>
<b>2017-07-26 15:14:30,414-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH,
FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct
plug status or plug is not available, , ]'}, log id: 1498b3c4</b><br>
2017-07-26 15:14:30,420-07 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID:
null, Call Stack: null, Custom Event ID: -1, Message: Execution of
power management status on Host ovirt6.j2noc.com using Proxy Host
ovirt1.j2noc.com and Fence Agent ilo4:10.144.254.89 failed.<br>
2017-07-26 15:14:30,420-07 WARN
[org.ovirt.engine.core.bll.pm.FenceAgentExecutor] (default task-52)
[4d72d360-2b92-43f6-b2df-e80ee305c622] Fence action failed using
proxy host '10.144.110.99', trying another proxy<br>
2017-07-26 15:14:30,740-07 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation
ID: null, Call Stack: null, Custom Event ID: -1, Message: Executing
power management status on Host ovirt6.j2noc.com using Proxy Host
ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89.<br>
2017-07-26 15:14:30,741-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] START,
FenceVdsVDSCommand(HostName = ovirt2.j2noc.com,
FenceVdsVDSCommandParameters:{runAsync='true',
hostId='91d8fa70-fd24-4530-90f7-982ff068230b',
targetVdsId='00000000-0000-0000-0000-000000000000', action='STATUS',
agent='FenceAgent:{id='null', hostId='null', order='1', type='ilo4',
ip='10.144.254.89', port='null', user='Administrator',
password='***', encryptOptions='false', options='power_wait=4'}',
policy='null'}), log id: 67d837da<br>
2017-07-26 15:14:30,898-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] FINISH,
FenceVdsVDSCommand, return: FenceOperationResult:{status='ERROR',
powerStatus='UNKNOWN', message='[Failed: Unable to obtain correct
plug status or plug is not available, , ]'}, log id: 67d837da<br>
2017-07-26 15:14:30,903-07 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-52) [4d72d360-2b92-43f6-b2df-e80ee305c622] EVENT_ID:
FENCE_OPERATION_USING_AGENT_AND_PROXY_FAILED(9,021), Correlation ID:
null, Call Stack: null, Custom Event ID: -1, Message: Execution of
power management status on Host ovirt6.j2noc.com using Proxy Host
ovirt2.j2noc.com and Fence Agent ilo4:10.144.254.89 failed.<br>
<br>
<br>
<br>
I'm not sure the right syntax for fence_ipmilan since even a "good"
host fails:<br>
<br>
<br>
[root@ovirt4 prod vdsm]# fence_ipmilan -a 10.144.254.87 -P -l
Administrator -p *** -o status -v chassis power status <br>
Executing: /usr/bin/ipmitool -I lanplus -H 10.144.254.87 -U
Administrator -P [set] -p 623 -L ADMINISTRATOR chassis power status<br>
<br>
1 Error: Unable to establish IPMI v2 / RMCP+ session<br>
<br>
<br>
Failed: Unable to obtain correct plug status or plug is not
available<br>
<br>
<br>
<br>
<br>
Any ideas on what the issue is?<br>
<br>
</body>
</html>
--------------DDA22B530AEEAC5C11E37F1E--
6 years, 9 months
problem while moving/copying disks: vdsm low level image copy failed
by Johan Bernhardsson
Hello,
We get this error message while moving or copying some of the disks on
our main cluster running 4.1.2 on centos7
This is shown in the engine:
VDSM vbgkvm02 command HSMGetAllTasksStatusesVDS failed: low level Image
copy failed
I can copy it inside the host. And i can use dd to copy. Haven't tried
to run qemu-img manually yet.
This is from vdsm.log on the host:
2017-07-28 09:07:22,741+0200 ERROR (tasks/6) [root] Job u'c82d4c53-
3eb4-405e-a2d5-c4c77519360e' failed (jobs:217)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/jobs.py", line 154, in
run
self._run()
File "/usr/share/vdsm/storage/sdm/api/copy_data.py", line 88, in _run
self._operation.wait_for_completion()
File "/usr/lib/python2.7/site-packages/vdsm/qemuimg.py", line 329, in
wait_for_completion
self.poll(timeout)
File "/usr/lib/python2.7/site-packages/vdsm/qemuimg.py", line 324, in
poll
self.error)
QImgError: cmd=['/usr/bin/taskset', '--cpu-list', '0-15',
'/usr/bin/nice', '-n', '19', '/usr/bin/ionice', '-c', '3',
'/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f',
'raw', u'/rhev/data-center/mnt/glusterSD/vbgsan02:_fs02/0924ff77-ef51-
43
5b-b90d-50bfbf2e8de7/images/750f4184-b852-4b00-94fc-
476f3f5b93c7/3fe43487-3302-4b34-865a-07c5c6aedbf2', '-O', 'raw',
u'/rhev/data-center/mnt/glusterSD/10.137.30.105:_fs03/5d47a297-a21f-
4587-bb7c-dd00d52010d5/images/750f4184-b852-4b00-94fc-
476f3f5b93c7/3fe43487-3302-4b34-865
a-07c5c6aedbf2'], ecode=1, stdout=, stderr=qemu-img: error while
reading sector 12197886: No data available
, message=None
The storage domains are all based on gluster. The storage domains that
we see this on is configured as dispersed volumes.
Found a way to "fix" the problem. And that is to run dd if=/dev/vda
of=/dev/null bs=1M inside the virtual guest. After that we can copy an
image or use storage livemigration.
Is this a gluster problem or an vdsm problem? Or could it be something
with qemu-img?
/Johan
6 years, 9 months
Yum error while installing host on Centos7
by Iurcev, Massimiliano
I installed ovirt-engine on Centos7. My target is to have a single host
machine.
During the installation of the (first and unique) host, I get an error:
Failed to install Host xxxx. Yum Non-fatal POSTIN scriptlet failure in rpm
package gtk2-2.24.28-8.el7.x86_64.
This error is followed by other errors:
Failed to install Host xxxx. Failed to execute stage 'Package
installation': One or more elements within Yum transaction failed.
and finally:
Host xxxx installation failed. Command returned failure code 1 during SSH
session 'root(a)xxxx.mydomain.it'.
All other libraries are yum-installed without problems.
6 years, 9 months
ovirt and mixed selinux
by Bill James
I was hoping to migrate my systems to using selinux gradually.
I added 3 new nodes with selinux in permissive mode.
Migration fails to any of the previous hosts that currently have selinux
disabled.
Is it an all or nothing deal? Obviously not easy to reboot all nodes at
once.
2017-07-28 09:35:43,616 ERROR (migsrc/8c566813) [virt.vm]
(vmId='8c566813-4bee-4f04-be23-c9fc10e1e1f2') unsupported configuration:
Unable to find security driver for model selinux (migration:265)
2017-07-28 09:35:43,641 ERROR (migsrc/8c566813) [virt.vm]
(vmId='8c566813-4bee-4f04-be23-c9fc10e1e1f2') Failed to migrate
(migration:405)
Traceback (most recent call last):
ovirt-engine-4.1.0.4-1.el7.centos.noarch
libselinux-utils-2.5-6.el7.x86_64
related: http://lists.ovirt.org/pipermail/users/2016-October/076878.html
6 years, 9 months
[SOLVED] Re: How to stop "Failed to stop image transfer session. Ticket does not exist for image" spam
by Richard Chan
Solved: I did not see the "Cancel" option in the disk page when the disk
upload is stalled. Apologies for the noise.
On Sat, Jul 29, 2017 at 2:57 PM, Richard Chan <richard(a)treeboxsolutions.com>
wrote:
> Using oVirt 4.1.3 - I have some failed disk uploads but the logs are being
> spammed by
>
>
> "Failed to stop image transfer session. Ticket does not exist for image"
>
> How to stop this? Thnaks!
>
>
> --
> Richard Chan
>
>
--
Richard Chan
6 years, 9 months
Chrome 59 still will not upload to imageio, firefox ok
by Richard Chan
oVirt 4.1.4
I have imageio proxy working with web ui on firefox.
On Chrome 59 I read bz# https://bugzilla.redhat.com/show_bug.cgi?id=1430598
and set "EnableCommonNameFallbackForLocalAnchors": true.
However the upload fails with
Make sure ovirt-imageio-proxy service is installed and configured, and
ovirt-engine's certificate is registered as a valid CA in the browser. The
certificate can be fetched from https://
<engine_url>/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA
In the terminal I see
:ERROR:cert_verify_proc_nss.cc(923)] CERT_PKIXVerifyCert for XXXXXXXX
failed err=-8172
for both the web UI and imageio proxy. Any suggestions?
--
Richard Chan
6 years, 9 months
