VDSM hang
by paf1＠email.cz
This is a multi-part message in MIME format.
--------------C04C9E032CD3A8F03847AFCD
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Hello everybody,
for production usage i'm testing ovirt with gluster.
All components seems to be running fine but whenever I'm testing huge
workload, then node freez. Not the main OS, but VDSM mgmt and attached
services, VMs eg.
*mgmt *
oVirt - 4.1.0.4
centos 7.3-1611
*nodes* ( installed from ovirt image
/"ovirt-node-ng-installer-ovirt-4.1-2017030804.iso" )/
OS Version:== RHEL - 7 - 3.1611.el7.centos
OS Description:== oVirt Node 4.1.0
Kernel Version:== 3.10.0 - 514.10.2.el7.x86_64
KVM Version:== 2.6.0 - 28.el7_3.3.1
LIBVIRT Version:== libvirt-2.0.0-10.el7_3.5
VDSM Version:== vdsm-4.19.4-1.el7.centos
SPICE Version:== 0.12.4 - 20.el7_3
GlusterFS Version:== glusterfs-3.8.9-1.el7 ( LVM thinprovisioning in
replica 2 - created from ovirt GUI )
concurently running
- huge import from export domain ( net workload )
- sequential write to VMs local disk ( gluster replica sequential workload )
- VMs database huge select ( random IOps )
- huge old snapshot delete ( random IOps )
In this configuration / workload is runnig one hour eg, with no
exceptions , with 70-80% disk load, but in some point VDSM freez all
jobs for a timeout and VM's are in "uknown" status .
The whole system revitalize then automaticaly in cca 20min time frame (
except the import and snapshot deleting(rollback) )
engine.log - focus 10:39:07 time ( Failed in
'HSMGetAllTasksStatusesVDS' method )
========
n child command id: 'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete
2017-03-10 10:39:01,727+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(DefaultQuartzScheduler2) [759c8e1f] Command
'RemoveSnapshotSingleDiskLive' (id:
'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id:
'33df2c1e-6ce3-44fd-a39b-d111883b4c4e' type:'DestroyImage' to complete
2017-03-10 10:39:03,929+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 446d0cd3
2017-03-10 10:39:04,343+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH,
GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED,
16.0.0.164:CONNECTED], log id: 446d0cd3
2017-03-10 10:39:04,353+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 69ea1fda
2017-03-10 10:39:05,128+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 69ea1fda
2017-03-10 10:39:07,163+01 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler2) [759c8e1f] Failed in
'HSMGetAllTasksStatusesVDS' method
2017-03-10 10:39:07,178+01 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler2) [759c8e1f] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: VDSM 2kvm2 command
HSMGetAllTasksStatusesVDS failed: Connection timed out
2017-03-10 10:39:07,182+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2)
[759c8e1f] BaseAsyncTask::onTaskEndSuccess: Task
'f594bf69-619b-4d1b-8f6d-a9826997e478' (Parent Command 'ImportVm',
Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended
successfully.
2017-03-10 10:39:07,182+01 INFO
[org.ovirt.engine.core.bll.CommandMultiAsyncTasks]
(DefaultQuartzScheduler2) [759c8e1f] Task with DB Task ID
'a05c7c07-9b98-4ab2-ac7b-9e70a75ba7b7' and VDSM Task ID
'7c60369f-70a3-4a6a-80c9-4753ac9ed372' is in state Polling. End action
for command 8deb3fe3-4a83-4605-816c-ffdc63fd9ac1 will proceed when all
the entity's tasks are completed.
2017-03-10 10:39:07,182+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2)
[759c8e1f] SPMAsyncTask::PollTask: Polling task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command 'DestroyImage',
Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned
status 'finished', result 'cleanSuccess'.
2017-03-10 10:39:07,182+01 ERROR
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler2)
[759c8e1f] BaseAsyncTask::logEndTaskFailure: Task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command 'DestroyImage',
Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended
with failure:
-- Result: 'cleanSuccess'
-- Message: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Connection timed out, code = 100',
-- Exception: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Connection timed out, code = 100'
2017-03-10 10:39:07,184+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f]
CommandAsyncTask::endActionIfNecessary: All tasks of command
'33df2c1e-6ce3-44fd-a39b-d111883b4c4e' has ended -> executing 'endAction'
2017-03-10 10:39:07,185+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f] CommandAsyncTask::endAction: Ending
action for '1' tasks (command ID:
'33df2c1e-6ce3-44fd-a39b-d111883b4c4e'): calling endAction '.
2017-03-10 10:39:07,185+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::endCommandAction [within thread] context: Attempting
to endAction 'DestroyImage',
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.storage.disk.image.DestroyImageCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] Command
[id=33df2c1e-6ce3-44fd-a39b-d111883b4c4e]: Updating status to 'FAILED',
The command end method logic will be executed by one of its parent commands.
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::HandleEndActionResult [within thread]: endAction for
action type 'DestroyImage' completed, handling the result.
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::HandleEndActionResult [within thread]: endAction for
action type 'DestroyImage' succeeded, clearing tasks.
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
SPMAsyncTask::ClearAsyncTask: Attempting to clear task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352'
2017-03-10 10:39:07,193+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] START,
SPMClearTaskVDSCommand(
SPMTaskGuidBaseVDSCommandParameters:{runAsync='true',
storagePoolId='00000001-0001-0001-0001-000000000311',
ignoreFailoverLimit='false',
taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id: 2b7080c2
2017-03-10 10:39:07,194+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] START,
HSMClearTaskVDSCommand(HostName = 2kvm2,
HSMTaskGuidBaseVDSCommandParameters:{runAsync='true',
hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8',
taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id: 2edff460
2017-03-10 10:39:08,208+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH,
HSMClearTaskVDSCommand, log id: 2edff460
2017-03-10 10:39:08,208+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH,
SPMClearTaskVDSCommand, log id: 2b7080c2
2017-03-10 10:39:08,213+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
BaseAsyncTask::removeTaskFromDB: Removed task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352' from DataBase
2017-03-10 10:39:08,213+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::HandleEndActionResult [within thread]: Removing
CommandMultiAsyncTasks object for entity
'33df2c1e-6ce3-44fd-a39b-d111883b4c4e'
2017-03-10 10:39:10,142+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2e7278cb
2017-03-10 10:39:11,513+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c]
FINISH, GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id: 2e7278cb
2017-03-10 10:39:11,523+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 43704ef2
2017-03-10 10:39:11,777+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler9) [67e1d8ed] Command 'RemoveSnapshot' (id:
'13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id:
'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete
2017-03-10 10:39:11,789+01 WARN
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(DefaultQuartzScheduler9) [759c8e1f] Child command 'DESTROY_IMAGE'
failed, proceeding to verify
2017-03-10 10:39:11,789+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(DefaultQuartzScheduler9) [759c8e1f] Executing Live Merge command step
'DESTROY_IMAGE_CHECK'
2017-03-10 10:39:11,832+01 INFO
[org.ovirt.engine.core.bll.DestroyImageCheckCommand] (pool-5-thread-7)
[4856f570] Running command: DestroyImageCheckCommand internal: true.
2017-03-10 10:39:11,833+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand]
(pool-5-thread-7) [4856f570] START, SPMGetVolumeInfoVDSCommand(
SPMGetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]',
runAsync='true', storagePoolId='00000001-0001-0001-0001-000000000311',
ignoreFailoverLimit='false',
storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e',
imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428',
imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id: 2c8031f8
2017-03-10 10:39:11,833+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand]
(pool-5-thread-7) [4856f570] Executing GetVolumeInfo using the current SPM
2017-03-10 10:39:11,834+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand]
(pool-5-thread-7) [4856f570] START, GetVolumeInfoVDSCommand(HostName =
2kvm2,
GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]',
runAsync='true', hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8',
storagePoolId='00000001-0001-0001-0001-000000000311',
storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e',
imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428',
imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id: 79ca86cc
2017-03-10 10:39:11,846+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler10) [a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c]
FINISH, GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 43704ef2
2017-03-10 10:39:16,858+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 6542adcd
2017-03-10 10:39:17,394+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH,
GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED,
16.0.0.164:CONNECTED], log id: 6542adcd
2017-03-10 10:39:17,406+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 44ec33ed
2017-03-10 10:39:18,598+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 44ec33ed
2017-03-10 10:39:21,865+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id:
'13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id:
'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete
2017-03-10 10:39:21,881+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(DefaultQuartzScheduler6) [4856f570] Command
'RemoveSnapshotSingleDiskLive' (id:
'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id:
'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc' type:'DestroyImageCheck' to complete
2017-03-10 10:39:23,611+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 4c2fc22d
2017-03-10 10:39:24,616+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] START,
GlusterTasksListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1f169371
2017-03-10 10:39:24,618+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] FINISH,
GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED,
16.0.0.164:CONNECTED], log id: 4c2fc22d
2017-03-10 10:39:24,629+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2ac55735
2017-03-10 10:39:24,822+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f] FINISH,
GlusterTasksListVDSCommand, return: [], log id: 1f169371
2017-03-10 10:39:26,836+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 2ac55735
2017-03-10 10:39:31,849+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 2e8dbcd1
2017-03-10 10:39:31,932+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id:
'13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id:
'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete
2017-03-10 10:39:31,944+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(DefaultQuartzScheduler6) [4856f570] Command
'RemoveSnapshotSingleDiskLive' (id:
'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command id:
'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc' type:'DestroyImageCheck' to complete
2017-03-10 10:39:33,213+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH,
GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED,
16.0.0.164:CONNECTED], log id: 2e8dbcd1
2017-03-10 10:39:33,226+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1fb3f9e3
2017-03-10 10:39:34,375+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 1fb3f9e3
2017-03-10 10:39:39,392+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 1e0b8eeb
2017-03-10 10:39:40,753+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] FINISH,
GlusterServersListVDSCommand, return: [172.16.5.163/24:CONNECTED,
16.0.0.164:CONNECTED], log id: 1e0b8eeb
2017-03-10 10:39:40,763+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id: 35b04b33
2017-03-10 10:39:41,952+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 35b04b33
2017-03-10 10:39:41,991+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot' (id:
'13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child command id:
'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete
gluster ( nothing in logs )
======
## "etc-glusterfs-glusterd.vol.log"
[2017-03-10 10:13:52.599019] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:16:48.639635] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
The message "I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1" repeated 3 times between
[2017-03-10 10:16:48.639635] and [2017-03-10 10:17:55.659379]
[2017-03-10 10:18:56.875516] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:19:57.204689] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:21:56.576879] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:21:57.772857] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:24:00.617931] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:30:04.918080] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:31:06.128638] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:32:07.325672] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:32:12.433586] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:32:13.544909] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:35:10.039213] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:37:19.905314] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:37:20.174209] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:38:12.635460] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
[2017-03-10 10:40:14.169864] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume slow1
## "rhev-data-center-mnt-glusterSD-localhost:_slow1.log"
[2017-03-10 09:43:40.346785] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61849:
dentry not found in 43e6968f-9c2a-40d8-8074-caf1a36f60cf
[2017-03-10 09:43:40.347076] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61879:
dentry not found in 902a6e3d-b7aa-439f-8262-cdc1b7f9f022
[2017-03-10 09:43:40.347145] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61935:
dentry not found in 846bbcfc-f2b3-4ab6-af44-aeaa10b39318
[2017-03-10 09:43:40.347211] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61922:
dentry not found in 66ad3bc5-26c7-4360-b33b-a084e3305cf8
[2017-03-10 09:43:40.351571] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61834:
dentry not found in 3b8278e1-40e5-4363-b21e-7bffcd024c62
[2017-03-10 09:43:40.352449] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61870:
dentry not found in 282f4c05-e09a-48e0-96a3-52e079ff2f73
[2017-03-10 09:50:38.829325] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)
[2017-03-10 09:50:42.221775] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)
[2017-03-10 09:50:45.956432] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)
[2017-03-10 09:50:40.349563] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)
[2017-03-10 09:50:44.503866] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)
[2017-03-10 09:59:46.860762] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.6684:
dentry not found in d1e65eea-8758-4407-ac2e-3605dc661364
[2017-03-10 10:02:22.500865] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.8767:
dentry not found in e228bb28-9602-4f8e-8323-7434d77849fc
[2017-03-10 10:04:03.103839] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.9787:
dentry not found in 6be71632-aa36-4975-b673-1357e0355027
[2017-03-10 10:06:02.406385] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)
... no other record
messages
========
several times occured:
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported value for cmd.
Mar 10 09:10:01 2kvm2 systemd: Started Session 274 of user root.
Mar 10 09:10:01 2kvm2 systemd: Starting Session 274 of user root.
Mar 10 09:20:02 2kvm2 systemd: Started Session 275 of user root.
Mar 10 09:20:02 2kvm2 systemd: Starting Session 275 of user root.
Mar 10 09:22:59 2kvm2 sanlock[1673]: 2017-03-10 09:22:59+0100 136031
[2576]: s3 delta_renew long write time 11 sec
Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure: order:2,
mode:0x104020
Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1 Tainted:
G I ------------ 3.10.0-514.10.2.el7.x86_64 #1
Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro
X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014
Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9
ffff88301f4839d8 ffffffff816864ef
Mar 10 09:24:03 2kvm2 kernel: ffff88301f483a68 ffffffff81186ba0
000068fc00000000 0000000000000000
Mar 10 09:24:03 2kvm2 kernel: fffffffffffffffc 0010402000000000
ffff88301567ae80 00000000f7228dc9
Mar 10 09:24:03 2kvm2 kernel: Call Trace:
Mar 10 09:24:03 2kvm2 kernel: <IRQ> [<ffffffff816864ef>]
dump_stack+0x19/0x1b
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81186ba0>]
warn_alloc_failed+0x110/0x180
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81682083>]
__alloc_pages_slowpath+0x6b7/0x725
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8118b155>]
__alloc_pages_nodemask+0x405/0x420
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811cf30a>]
alloc_pages_current+0xaa/0x170
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81185a7e>] __get_free_pages+0xe/0x50
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dabae>]
kmalloc_order_trace+0x2e/0xa0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dd381>] __kmalloc+0x221/0x240
Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f83fa>]
bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x]
Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f92f7>]
bnx2x_rx_int+0x227/0x17b0 [bnx2x]
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81033669>] ? sched_clock+0x9/0x10
Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02fc72d>]
bnx2x_poll+0x1dd/0x260 [bnx2x]
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff815705e0>] net_rx_action+0x170/0x380
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f2cf>] __do_softirq+0xef/0x280
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8169859c>] call_softirq+0x1c/0x30
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8102d365>] do_softirq+0x65/0xa0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f665>] irq_exit+0x115/0x120
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81699138>] do_IRQ+0x58/0xf0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8168e2ad>]
common_interrupt+0x6d/0x6d
Mar 10 09:24:03 2kvm2 kernel: <EOI> [<ffffffff81189a73>] ?
free_hot_cold_page+0x103/0x160
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81189b16>]
free_hot_cold_page_list+0x46/0xa0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195193>]
shrink_page_list+0x543/0xb00
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195dda>]
shrink_inactive_list+0x1fa/0x630
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196975>] shrink_lruvec+0x385/0x770
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196dd6>] shrink_zone+0x76/0x1a0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8119807c>] balance_pgdat+0x48c/0x5e0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81198343>] kswapd+0x173/0x450
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b17d0>] ?
wake_up_atomic_t+0x30/0x30
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811981d0>] ?
balance_pgdat+0x5e0/0x5e0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b06ff>] kthread+0xcf/0xe0
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ?
kthread_create_on_node+0x140/0x140
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81696a58>] ret_from_fork+0x58/0x90
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ?
kthread_create_on_node+0x140/0x140
Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure: order:2,
mode:0x104020
Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1 Tainted:
G I ------------ 3.10.0-514.10.2.el7.x86_64 #1
Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro
X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014
Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9
ffff88301f4839d8 ffffffff816864ef
and in critical time again
Mar 10 10:37:53 2kvm2 sanlock[1673]: 2017-03-10 10:37:53+0100 140524
[1673]: s3 check_our_lease warning 73 last_success 140451
Mar 10 10:37:54 2kvm2 sanlock[1673]: 2017-03-10 10:37:54+0100 140525
[1673]: s3 check_our_lease warning 74 last_success 140451
Mar 10 10:37:54 2kvm2 wdmd[1732]: test warning now 140526 ping 140516
close 0 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1
Mar 10 10:37:54 2kvm2 kernel: watchdog watchdog0: watchdog did not stop!
Mar 10 10:37:54 2kvm2 wdmd[1732]: /dev/watchdog0 closed unclean
Mar 10 10:37:55 2kvm2 sanlock[1673]: 2017-03-10 10:37:55+0100 140526
[1673]: s3 check_our_lease warning 75 last_success 140451
Mar 10 10:37:55 2kvm2 wdmd[1732]: test warning now 140527 ping 140516
close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1
Mar 10 10:37:56 2kvm2 sanlock[1673]: 2017-03-10 10:37:56+0100 140527
[1673]: s3 check_our_lease warning 76 last_success 140451
Mar 10 10:37:56 2kvm2 wdmd[1732]: test warning now 140528 ping 140516
close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1
Mar 10 10:37:57 2kvm2 sanlock[1673]: 2017-03-10 10:37:57+0100 140528
[1673]: s3 check_our_lease warning 77 last_success 140451
Mar 10 10:37:57 2kvm2 wdmd[1732]: test warning now 140529 ping 140516
close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1
Mar 10 10:37:58 2kvm2 sanlock[1673]: 2017-03-10 10:37:58+0100 140529
[1673]: s3 check_our_lease warning 78 last_success 140451
Mar 10 10:37:58 2kvm2 wdmd[1732]: test warning now 140530 ping 140516
close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1
Mar 10 10:37:59 2kvm2 sanlock[1673]: 2017-03-10 10:37:59+0100 140530
[1673]: s3 check_our_lease warning 79 last_success 140451
Mar 10 10:37:59 2kvm2 wdmd[1732]: test failed rem 55 now 140531 ping
140516 close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1
Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100 140531
[1673]: s3 check_our_lease failed 80
Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100 140531
[1673]: s3 all pids clear
Mar 10 10:38:01 2kvm2 wdmd[1732]: /dev/watchdog0 reopen
Mar 10 10:38:10 2kvm2 journal: Cannot start job (query, none) for domain
TEST-LBS_EBSAPP; current job is (query, none) owned by (3284
remoteDispatchConnectGetAllDomainStats, 0 <null>) for (62s, 0s)
Mar 10 10:38:10 2kvm2 journal: Timed out during operation: cannot
acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)
Mar 10 10:38:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error
receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1',
40590, 0, 0) at 0x3acdd88>: unexpected eof
Mar 10 10:38:40 2kvm2 journal: Cannot start job (query, none) for domain
TEST1-LBS_ATRYA; current job is (query, none) owned by (3288
remoteDispatchConnectGetAllDomainStats, 0 <null>) for (47s, 0s)
Mar 10 10:38:40 2kvm2 journal: Timed out during operation: cannot
acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)
Mar 10 10:38:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error
receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1',
40592, 0, 0) at 0x3fd5b90>: unexpected eof
Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none) for domain
TEST-LBS_EBSAPP; current job is (query, none) owned by (3284
remoteDispatchConnectGetAllDomainStats, 0 <null>) for (122s, 0s)
Mar 10 10:39:10 2kvm2 journal: Timed out during operation: cannot
acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)
Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none) for domain
TEST1-LBS_ATRYA; current job is (query, none) owned by (3288
remoteDispatchConnectGetAllDomainStats, 0 <null>) for (77s, 0s)
Mar 10 10:39:10 2kvm2 journal: Timed out during operation: cannot
acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)
Mar 10 10:39:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error
receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1',
40594, 0, 0) at 0x2447290>: unexpected eof
Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100 140615
[2576]: s3 delta_renew write time 140 error -202
Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100 140615
[2576]: s3 renewal error -202 delta_length 144 last_success 140451
Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none) for domain
TEST-LBS_EBSAPP; current job is (query, none) owned by (3284
remoteDispatchConnectGetAllDomainStats, 0 <null>) for (152s, 0s)
Mar 10 10:39:40 2kvm2 journal: Timed out during operation: cannot
acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)
Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none) for domain
TEST1-LBS_ATRYA; current job is (query, none) owned by (3288
remoteDispatchConnectGetAllDomainStats, 0 <null>) for (107s, 0s)
Mar 10 10:39:40 2kvm2 journal: Timed out during operation: cannot
acquire state change lock (held by remoteDispatchConnectGetAllDomainStats)
Mar 10 10:39:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL error
receiving from <yajsonrpc.betterAsyncore.Dispatcher connected ('::1',
40596, 0, 0) at 0x2472ef0>: unexpected eof
Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42107 blocked for more
than 120 seconds.
Mar 10 10:39:49 2kvm2 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 10:39:49 2kvm2 kernel: qemu-img D ffff88010dad3e30 0
42107 3631 0x00000080
Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3b30 0000000000000082
ffff8814491f4e70 ffff88010dad3fd8
Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3fd8 ffff88010dad3fd8
ffff8814491f4e70 ffff88301f096c40
Mar 10 10:39:49 2kvm2 kernel: 0000000000000000 7fffffffffffffff
ffff88181f186c00 ffff88010dad3e30
Mar 10 10:39:49 2kvm2 kernel: Call Trace:
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168bbb9>] schedule+0x29/0x70
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81689609>]
schedule_timeout+0x239/0x2d0
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b15e>]
io_schedule_timeout+0xae/0x130
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b1f8>] io_schedule+0x18/0x20
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8124d9e5>]
wait_on_sync_kiocb+0x35/0x80
Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36091>]
fuse_direct_IO+0x231/0x380 [fuse]
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a6ddd>] ?
cap_inode_need_killpriv+0x2d/0x40
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a8cb6>] ?
security_inode_need_killpriv+0x16/0x20
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81219e3f>] ?
dentry_needs_remove_privs.part.13+0x1f/0x30
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81182a2d>]
generic_file_direct_write+0xcd/0x190
Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36905>]
fuse_file_aio_write+0x185/0x340 [fuse]
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fdabd>] do_sync_write+0x8d/0xd0
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fe32d>] vfs_write+0xbd/0x1e0
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811ff002>] SyS_pwrite64+0x92/0xc0
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81696b09>]
system_call_fastpath+0x16/0x1b
Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42111 blocked for more
than 120 seconds.
Mar 10 10:39:49 2kvm2 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 10:39:49 2kvm2 kernel: qemu-img D ffff8818a76e7e30 0
42111 3632 0x00000080
Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7b30 0000000000000082
ffff88188aaeaf10 ffff8818a76e7fd8
Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7fd8 ffff8818a76e7fd8
ffff88188aaeaf10 ffff88301f156c40
memory
=======
# cat /proc/meminfo
MemTotal: 197983472 kB
MemFree: 834228 kB
MemAvailable: 165541204 kB
Buffers: 45548 kB
Cached: 159596272 kB
SwapCached: 119872 kB
Active: 40803264 kB
Inactive: 148022076 kB
Active(anon): 26594112 kB
Inactive(anon): 2626384 kB
Active(file): 14209152 kB
Inactive(file): 145395692 kB
Unevictable: 50488 kB
Mlocked: 50488 kB
SwapTotal: 4194300 kB
SwapFree: 3612188 kB
Dirty: 624 kB
Writeback: 0 kB
AnonPages: 29185032 kB
Mapped: 85176 kB
Shmem: 25908 kB
Slab: 6203384 kB
SReclaimable: 5857240 kB
SUnreclaim: 346144 kB
KernelStack: 19184 kB
PageTables: 86100 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 103186036 kB
Committed_AS: 52300288 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1560580 kB
VmallocChunk: 34257341440 kB
HardwareCorrupted: 0 kB
AnonHugePages: 5566464 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 431292 kB
DirectMap2M: 19382272 kB
DirectMap1G: 183500800 kB
can anybody help me with this ??
I've got a small tip about swap problem ( in messages), but not shure .....
The similar problem occured in older versions in gluster/ovirt testing (
in huge workload freeez - but not fatal overload )
regards
Paf1
--------------C04C9E032CD3A8F03847AFCD
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000066">
<font face="Ubuntu">Hello everybody,<br>
<br>
for production usage i'm testing ovirt with gluster.<br>
All components seems to be running fine but whenever I'm testing
huge workload, then node freez. Not the main OS, but VDSM mgmt and
attached services, VMs eg.<br>
<br>
<b>mgmt </b><br>
oVirt - 4.1.0.4<br>
centos 7.3-1611<br>
<br>
<br>
<b>nodes</b> ( installed from ovirt image <i>"ovirt-node-ng-installer-ovirt-4.1-2017030804.iso"
)</i><br>
</font><br>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">OS Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row0_value"
class="GOJECEMBACD"> == RHEL - 7 - 3.1611.el7.centos</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">OS Description:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row1_value"
class="GOJECEMBACD">== oVirt Node 4.1.0</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">Kernel Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row2_value"
class="GOJECEMBACD">== 3.10.0 - 514.10.2.el7.x86_64</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">KVM Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row3_value"
class="GOJECEMBACD">== 2.6.0 - 28.el7_3.3.1</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">LIBVIRT Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row4_value"
class="GOJECEMBACD">== libvirt-2.0.0-10.el7_3.5</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">VDSM Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row5_value"
class="GOJECEMBACD">== vdsm-4.19.4-1.el7.centos</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">SPICE Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row6_value"
class="GOJECEMBACD">== 0.12.4 - 20.el7_3</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">GlusterFS Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row7_value"
class="GOJECEMBACD">== glusterfs-3.8.9-1.el7 ( LVM
thinprovisioning in replica 2 - created from ovirt GUI )<br>
</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="GOJECEMBPBD"
id="SubTabHostGeneralSoftwareView_formPanel_col0_row8_label"><br>
</div>
</div>
</div>
</div>
</div>
<font face="Ubuntu">concurently running<br>
- huge import from export domain ( net workload )<br>
- sequential write to VMs local disk ( gluster replica sequential
workload )<br>
- VMs database huge select ( random IOps )<br>
- huge old snapshot delete ( random IOps )<br>
<br>
In this configuration / workload is runnig one hour eg, with no
exceptions , with 70-80% disk load, but in some point VDSM freez
all jobs for a timeout and VM's are in "uknown" status .<br>
The whole system revitalize then automaticaly in cca 20min time
frame ( except the import and snapshot deleting(rollback) )<br>
<br>
engine.log - focus 10:39:07 time ( </font><font face="Ubuntu"><font
face="Ubuntu">Failed in 'HSMGetAllTasksStatusesVDS' method )<br>
</font>========<br>
<br>
<font size="-1">n child command id:
'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete<br>
2017-03-10 10:39:01,727+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(DefaultQuartzScheduler2) [759c8e1f] Command
'RemoveSnapshotSingleDiskLive' (id:
'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command
id: '33df2c1e-6ce3-44fd-a39b-d111883b4c4e' type:'DestroyImage'
to complete<br>
2017-03-10 10:39:03,929+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4]
START, GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
446d0cd3<br>
2017-03-10 10:39:04,343+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4]
FINISH, GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id:
446d0cd3<br>
2017-03-10 10:39:04,353+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4]
START, GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
69ea1fda<br>
2017-03-10 10:39:05,128+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler5) [fde51205-3e8b-4b84-a478-352dc444ccc4]
FINISH, GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 69ea1fda<br>
2017-03-10 10:39:07,163+01 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler2) [759c8e1f] Failed in
'HSMGetAllTasksStatusesVDS' method<br>
2017-03-10 10:39:07,178+01 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler2) [759c8e1f] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call
Stack: null, Custom Event ID: -1, Message: VDSM 2kvm2 command
HSMGetAllTasksStatusesVDS failed: Connection timed out<br>
2017-03-10 10:39:07,182+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f]
BaseAsyncTask::onTaskEndSuccess: Task
'f594bf69-619b-4d1b-8f6d-a9826997e478' (Parent Command
'ImportVm', Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.<br>
2017-03-10 10:39:07,182+01 INFO
[org.ovirt.engine.core.bll.CommandMultiAsyncTasks]
(DefaultQuartzScheduler2) [759c8e1f] Task with DB Task ID
'a05c7c07-9b98-4ab2-ac7b-9e70a75ba7b7' and VDSM Task ID
'7c60369f-70a3-4a6a-80c9-4753ac9ed372' is in state Polling. End
action for command 8deb3fe3-4a83-4605-816c-ffdc63fd9ac1 will
proceed when all the entity's tasks are completed.<br>
2017-03-10 10:39:07,182+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f] SPMAsyncTask::PollTask:
Polling task 'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent
Command 'DestroyImage', Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
returned status 'finished', result 'cleanSuccess'.<br>
2017-03-10 10:39:07,182+01 ERROR
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f]
BaseAsyncTask::logEndTaskFailure: Task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352' (Parent Command
'DestroyImage', Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended with failure:<br>
-- Result: 'cleanSuccess'<br>
-- Message: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Connection timed out, code =
100',<br>
-- Exception: 'VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = Connection timed out, code =
100'<br>
2017-03-10 10:39:07,184+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f]
CommandAsyncTask::endActionIfNecessary: All tasks of command
'33df2c1e-6ce3-44fd-a39b-d111883b4c4e' has ended -> executing
'endAction'<br>
2017-03-10 10:39:07,185+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(DefaultQuartzScheduler2) [759c8e1f]
CommandAsyncTask::endAction: Ending action for '1' tasks
(command ID: '33df2c1e-6ce3-44fd-a39b-d111883b4c4e'): calling
endAction '.<br>
2017-03-10 10:39:07,185+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::endCommandAction [within thread] context:
Attempting to endAction 'DestroyImage',<br>
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.storage.disk.image.DestroyImageCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] Command
[id=33df2c1e-6ce3-44fd-a39b-d111883b4c4e]: Updating status to
'FAILED', The command end method logic will be executed by one
of its parent commands.<br>
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::HandleEndActionResult [within thread]:
endAction for action type 'DestroyImage' completed, handling the
result.<br>
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::HandleEndActionResult [within thread]:
endAction for action type 'DestroyImage' succeeded, clearing
tasks.<br>
2017-03-10 10:39:07,192+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
SPMAsyncTask::ClearAsyncTask: Attempting to clear task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352'<br>
2017-03-10 10:39:07,193+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] START,
SPMClearTaskVDSCommand(
SPMTaskGuidBaseVDSCommandParameters:{runAsync='true',
storagePoolId='00000001-0001-0001-0001-000000000311',
ignoreFailoverLimit='false',
taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id:
2b7080c2<br>
2017-03-10 10:39:07,194+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] START,
HSMClearTaskVDSCommand(HostName = 2kvm2,
HSMTaskGuidBaseVDSCommandParameters:{runAsync='true',
hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8',
taskId='f351e8f6-6dd7-49aa-bf54-650d84fc6352'}), log id:
2edff460<br>
2017-03-10 10:39:08,208+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH,
HSMClearTaskVDSCommand, log id: 2edff460<br>
2017-03-10 10:39:08,208+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f] FINISH,
SPMClearTaskVDSCommand, log id: 2b7080c2<br>
2017-03-10 10:39:08,213+01 INFO
[org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
BaseAsyncTask::removeTaskFromDB: Removed task
'f351e8f6-6dd7-49aa-bf54-650d84fc6352' from DataBase<br>
2017-03-10 10:39:08,213+01 INFO
[org.ovirt.engine.core.bll.tasks.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-31) [759c8e1f]
CommandAsyncTask::HandleEndActionResult [within thread]:
Removing CommandMultiAsyncTasks object for entity
'33df2c1e-6ce3-44fd-a39b-d111883b4c4e'<br>
2017-03-10 10:39:10,142+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler10)
[a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
2e7278cb<br>
2017-03-10 10:39:11,513+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler10)
[a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] FINISH,
GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id:
2e7278cb<br>
2017-03-10 10:39:11,523+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler10)
[a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
43704ef2<br>
2017-03-10 10:39:11,777+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler9) [67e1d8ed] Command 'RemoveSnapshot'
(id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child
command id: 'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete<br>
2017-03-10 10:39:11,789+01 WARN
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(DefaultQuartzScheduler9) [759c8e1f] Child command
'DESTROY_IMAGE' failed, proceeding to verify<br>
2017-03-10 10:39:11,789+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(DefaultQuartzScheduler9) [759c8e1f] Executing Live Merge
command step 'DESTROY_IMAGE_CHECK'<br>
2017-03-10 10:39:11,832+01 INFO
[org.ovirt.engine.core.bll.DestroyImageCheckCommand]
(pool-5-thread-7) [4856f570] Running command:
DestroyImageCheckCommand internal: true.<br>
2017-03-10 10:39:11,833+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand]
(pool-5-thread-7) [4856f570] START, SPMGetVolumeInfoVDSCommand(
SPMGetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]',
runAsync='true',
storagePoolId='00000001-0001-0001-0001-000000000311',
ignoreFailoverLimit='false',
storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e',
imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428',
imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id:
2c8031f8<br>
2017-03-10 10:39:11,833+01 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand]
(pool-5-thread-7) [4856f570] Executing GetVolumeInfo using the
current SPM<br>
2017-03-10 10:39:11,834+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand]
(pool-5-thread-7) [4856f570] START,
GetVolumeInfoVDSCommand(HostName = 2kvm2,
GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]',
runAsync='true', hostId='905375e1-6de4-4fdf-b69c-b2d546f869c8',
storagePoolId='00000001-0001-0001-0001-000000000311',
storageDomainId='1603cd90-92ef-4c03-922c-cecb282fd00e',
imageGroupId='7543338a-3ca6-4698-bb50-c14f0bd71428',
imageId='50b592f7-bfba-4398-879c-8d6a19a2c000'}), log id:
79ca86cc<br>
2017-03-10 10:39:11,846+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler10)
[a86dc7b5-52dc-40d4-a3b9-49d7eabbb93c] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 43704ef2<br>
2017-03-10 10:39:16,858+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f]
START, GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
6542adcd<br>
2017-03-10 10:39:17,394+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f]
FINISH, GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id:
6542adcd<br>
2017-03-10 10:39:17,406+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f]
START, GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
44ec33ed<br>
2017-03-10 10:39:18,598+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f]
FINISH, GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 44ec33ed<br>
2017-03-10 10:39:21,865+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot'
(id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child
command id: 'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete<br>
2017-03-10 10:39:21,881+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(DefaultQuartzScheduler6) [4856f570] Command
'RemoveSnapshotSingleDiskLive' (id:
'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command
id: 'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc'
type:'DestroyImageCheck' to complete<br>
2017-03-10 10:39:23,611+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] START,
GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
4c2fc22d<br>
2017-03-10 10:39:24,616+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f]
START, GlusterTasksListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
1f169371<br>
2017-03-10 10:39:24,618+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] FINISH,
GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id:
4c2fc22d<br>
2017-03-10 10:39:24,629+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] START,
GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
2ac55735<br>
2017-03-10 10:39:24,822+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(DefaultQuartzScheduler7) [d82701d9-9fa3-467d-b273-f5fe5a93062f]
FINISH, GlusterTasksListVDSCommand, return: [], log id: 1f169371<br>
2017-03-10 10:39:26,836+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler6) [4856f570] FINISH,
GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 2ac55735<br>
2017-03-10 10:39:31,849+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4]
START, GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
2e8dbcd1<br>
2017-03-10 10:39:31,932+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot'
(id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child
command id: 'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete<br>
2017-03-10 10:39:31,944+01 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(DefaultQuartzScheduler6) [4856f570] Command
'RemoveSnapshotSingleDiskLive' (id:
'a8a3a4d5-cf7d-4423-8243-022911232508') waiting on child command
id: 'b1d63b8e-19d3-4d64-8fa8-4eb3e2d1a8fc'
type:'DestroyImageCheck' to complete<br>
2017-03-10 10:39:33,213+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4]
FINISH, GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id:
2e8dbcd1<br>
2017-03-10 10:39:33,226+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4]
START, GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
1fb3f9e3<br>
2017-03-10 10:39:34,375+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler8) [fde51205-3e8b-4b84-a478-352dc444ccc4]
FINISH, GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 1fb3f9e3<br>
2017-03-10 10:39:39,392+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6]
START, GlusterServersListVDSCommand(HostName = 2kvm1,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
1e0b8eeb<br>
2017-03-10 10:39:40,753+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6]
FINISH, GlusterServersListVDSCommand, return:
[172.16.5.163/24:CONNECTED, 16.0.0.164:CONNECTED], log id:
1e0b8eeb<br>
2017-03-10 10:39:40,763+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6]
START, GlusterVolumesListVDSCommand(HostName = 2kvm1,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='86876b79-71d8-4ae1-883b-ba010cd270e7'}), log id:
35b04b33<br>
2017-03-10 10:39:41,952+01 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler9) [12d6d15f-e054-4833-bd87-58f6a51e5fa6]
FINISH, GlusterVolumesListVDSCommand, return:
{8ded4083-2f31-489e-a60d-a315a5eb9b22=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7765e4ad},
log id: 35b04b33<br>
2017-03-10 10:39:41,991+01 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(DefaultQuartzScheduler6) [67e1d8ed] Command 'RemoveSnapshot'
(id: '13c2cb7c-0809-4971-aceb-37ae66105ab7') waiting on child
command id: 'a8a3a4d5-cf7d-4423-8243-022911232508'
type:'RemoveSnapshotSingleDiskLive' to complete<br>
<br>
</font><br>
gluster ( nothing in logs )<br>
======<br>
<br>
<br>
</font><font face="Ubuntu">## "etc-glusterfs-glusterd.vol.log"</font><font
face="Ubuntu"><font size="-1"><br>
[2017-03-10 10:13:52.599019] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:16:48.639635] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
The message "I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1"
repeated 3 times between [2017-03-10 10:16:48.639635] and
[2017-03-10 10:17:55.659379]<br>
[2017-03-10 10:18:56.875516] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:19:57.204689] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:21:56.576879] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:21:57.772857] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:24:00.617931] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:30:04.918080] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:31:06.128638] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:32:07.325672] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:32:12.433586] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:32:13.544909] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:35:10.039213] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:37:19.905314] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:37:20.174209] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:38:12.635460] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1<br>
[2017-03-10 10:40:14.169864] I [MSGID: 106499]
[glusterd-handler.c:4349:__glusterd_handle_status_volume]
0-management: Received status volume req for volume slow1</font><br>
<br>
<br>
## "rhev-data-center-mnt-glusterSD-localhost:_slow1.log"<br>
<font size="-1">[2017-03-10 09:43:40.346785] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61849:
dentry not found in 43e6968f-9c2a-40d8-8074-caf1a36f60cf<br>
[2017-03-10 09:43:40.347076] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61879:
dentry not found in 902a6e3d-b7aa-439f-8262-cdc1b7f9f022<br>
[2017-03-10 09:43:40.347145] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61935:
dentry not found in 846bbcfc-f2b3-4ab6-af44-aeaa10b39318<br>
[2017-03-10 09:43:40.347211] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61922:
dentry not found in 66ad3bc5-26c7-4360-b33b-a084e3305cf8<br>
[2017-03-10 09:43:40.351571] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61834:
dentry not found in 3b8278e1-40e5-4363-b21e-7bffcd024c62<br>
[2017-03-10 09:43:40.352449] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/b6f2d08d-2441-4111-ab62-e14abdfaf602.61870:
dentry not found in 282f4c05-e09a-48e0-96a3-52e079ff2f73<br>
[2017-03-10 09:50:38.829325] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)<br>
[2017-03-10 09:50:42.221775] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)<br>
[2017-03-10 09:50:45.956432] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/3cef54b4-45b9-4f5b-82c2-fcc8def06a37/85287865-38f0-45df-9e6c-1294913cbb88.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)<br>
[2017-03-10 09:50:40.349563] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/014ca3aa-d5f5-4b88-8f84-be8d4c5dfc1e/f147532a-89fa-49e0-8225-f82343fca8be.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)<br>
[2017-03-10 09:50:44.503866] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/4cf7dd90-9dcc-428c-82bc-fbf08dbee0be/12812d56-1606-4bf8-a391-0a2cacbd020b.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)<br>
[2017-03-10 09:59:46.860762] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.6684:
dentry not found in d1e65eea-8758-4407-ac2e-3605dc661364<br>
[2017-03-10 10:02:22.500865] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.8767:
dentry not found in e228bb28-9602-4f8e-8323-7434d77849fc<br>
[2017-03-10 10:04:03.103839] W [MSGID: 101159]
[inode.c:1214:__inode_unlink] 0-inode:
be318638-e8a0-4c6d-977d-7a937aa84806/6e105aa3-a3fc-4aca-be50-78b7642c4072.9787:
dentry not found in 6be71632-aa36-4975-b673-1357e0355027<br>
[2017-03-10 10:06:02.406385] I [MSGID: 109066]
[dht-rename.c:1569:dht_rename] 0-slow1-dht: renaming
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta.new
(hash=slow1-replicate-0/cache=slow1-replicate-0) =>
/1603cd90-92ef-4c03-922c-cecb282fd00e/images/2a9c1c6a-f045-4dce-a47b-95a2267eef72/6f264695-0669-4b49-a2f6-e6c92482f2fb.meta
(hash=slow1-replicate-0/cache=slow1-replicate-0)</font><br>
<font size="-1">... no other record</font><br>
<br>
<br>
messages<br>
========<br>
<br>
several times occured:<br>
<br>
<font size="-1">Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring
unsupported value for cmd.<br>
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported
value for cmd.<br>
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported
value for cmd.<br>
Mar 10 09:04:38 2kvm2 lvmetad: WARNING: Ignoring unsupported
value for cmd.<br>
Mar 10 09:10:01 2kvm2 systemd: Started Session 274 of user root.<br>
Mar 10 09:10:01 2kvm2 systemd: Starting Session 274 of user
root.<br>
Mar 10 09:20:02 2kvm2 systemd: Started Session 275 of user root.<br>
Mar 10 09:20:02 2kvm2 systemd: Starting Session 275 of user
root.<br>
Mar 10 09:22:59 2kvm2 sanlock[1673]: 2017-03-10 09:22:59+0100
136031 [2576]: s3 delta_renew long write time 11 sec<br>
Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure:
order:2, mode:0x104020<br>
Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1
Tainted: G I ------------
3.10.0-514.10.2.el7.x86_64 #1<br>
Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro
X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014<br>
Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9
ffff88301f4839d8 ffffffff816864ef<br>
Mar 10 09:24:03 2kvm2 kernel: ffff88301f483a68 ffffffff81186ba0
000068fc00000000 0000000000000000<br>
Mar 10 09:24:03 2kvm2 kernel: fffffffffffffffc 0010402000000000
ffff88301567ae80 00000000f7228dc9<br>
Mar 10 09:24:03 2kvm2 kernel: Call Trace:<br>
Mar 10 09:24:03 2kvm2 kernel: <IRQ>
[<ffffffff816864ef>] dump_stack+0x19/0x1b<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81186ba0>]
warn_alloc_failed+0x110/0x180<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81682083>]
__alloc_pages_slowpath+0x6b7/0x725<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8118b155>]
__alloc_pages_nodemask+0x405/0x420<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811cf30a>]
alloc_pages_current+0xaa/0x170<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81185a7e>]
__get_free_pages+0xe/0x50<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dabae>]
kmalloc_order_trace+0x2e/0xa0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811dd381>]
__kmalloc+0x221/0x240<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f83fa>]
bnx2x_frag_alloc.isra.62+0x2a/0x40 [bnx2x]<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02f92f7>]
bnx2x_rx_int+0x227/0x17b0 [bnx2x]<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81033669>] ?
sched_clock+0x9/0x10<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffffa02fc72d>]
bnx2x_poll+0x1dd/0x260 [bnx2x]<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff815705e0>]
net_rx_action+0x170/0x380<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f2cf>]
__do_softirq+0xef/0x280<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8169859c>]
call_softirq+0x1c/0x30<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8102d365>]
do_softirq+0x65/0xa0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8108f665>]
irq_exit+0x115/0x120<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81699138>]
do_IRQ+0x58/0xf0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8168e2ad>]
common_interrupt+0x6d/0x6d<br>
Mar 10 09:24:03 2kvm2 kernel: <EOI>
[<ffffffff81189a73>] ? free_hot_cold_page+0x103/0x160<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81189b16>]
free_hot_cold_page_list+0x46/0xa0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195193>]
shrink_page_list+0x543/0xb00<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81195dda>]
shrink_inactive_list+0x1fa/0x630<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196975>]
shrink_lruvec+0x385/0x770<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81196dd6>]
shrink_zone+0x76/0x1a0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff8119807c>]
balance_pgdat+0x48c/0x5e0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81198343>]
kswapd+0x173/0x450<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b17d0>] ?
wake_up_atomic_t+0x30/0x30<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff811981d0>] ?
balance_pgdat+0x5e0/0x5e0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b06ff>]
kthread+0xcf/0xe0<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ?
kthread_create_on_node+0x140/0x140<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff81696a58>]
ret_from_fork+0x58/0x90<br>
Mar 10 09:24:03 2kvm2 kernel: [<ffffffff810b0630>] ?
kthread_create_on_node+0x140/0x140<br>
Mar 10 09:24:03 2kvm2 kernel: kswapd1: page allocation failure:
order:2, mode:0x104020<br>
Mar 10 09:24:03 2kvm2 kernel: CPU: 42 PID: 265 Comm: kswapd1
Tainted: G I ------------
3.10.0-514.10.2.el7.x86_64 #1<br>
Mar 10 09:24:03 2kvm2 kernel: Hardware name: Supermicro
X10DRC/X10DRi-LN4+, BIOS 1.0a 08/29/2014<br>
Mar 10 09:24:03 2kvm2 kernel: 0000000000104020 00000000f7228dc9
ffff88301f4839d8 ffffffff816864ef<br>
</font><br>
<br>
and in critical time again<br>
<br>
<font size="-1">Mar 10 10:37:53 2kvm2 sanlock[1673]: 2017-03-10
10:37:53+0100 140524 [1673]: s3 check_our_lease warning 73
last_success 140451<br>
Mar 10 10:37:54 2kvm2 sanlock[1673]: 2017-03-10 10:37:54+0100
140525 [1673]: s3 check_our_lease warning 74 last_success 140451<br>
Mar 10 10:37:54 2kvm2 wdmd[1732]: test warning now 140526 ping
140516 close 0 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br>
Mar 10 10:37:54 2kvm2 kernel: watchdog watchdog0: watchdog did
not stop!<br>
Mar 10 10:37:54 2kvm2 wdmd[1732]: /dev/watchdog0 closed unclean<br>
Mar 10 10:37:55 2kvm2 sanlock[1673]: 2017-03-10 10:37:55+0100
140526 [1673]: s3 check_our_lease warning 75 last_success 140451<br>
Mar 10 10:37:55 2kvm2 wdmd[1732]: test warning now 140527 ping
140516 close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br>
Mar 10 10:37:56 2kvm2 sanlock[1673]: 2017-03-10 10:37:56+0100
140527 [1673]: s3 check_our_lease warning 76 last_success 140451<br>
Mar 10 10:37:56 2kvm2 wdmd[1732]: test warning now 140528 ping
140516 close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br>
Mar 10 10:37:57 2kvm2 sanlock[1673]: 2017-03-10 10:37:57+0100
140528 [1673]: s3 check_our_lease warning 77 last_success 140451<br>
Mar 10 10:37:57 2kvm2 wdmd[1732]: test warning now 140529 ping
140516 close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br>
Mar 10 10:37:58 2kvm2 sanlock[1673]: 2017-03-10 10:37:58+0100
140529 [1673]: s3 check_our_lease warning 78 last_success 140451<br>
Mar 10 10:37:58 2kvm2 wdmd[1732]: test warning now 140530 ping
140516 close 140526 renewal 140451 expire 140531 client 1673
sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br>
Mar 10 10:37:59 2kvm2 sanlock[1673]: 2017-03-10 10:37:59+0100
140530 [1673]: s3 check_our_lease warning 79 last_success 140451<br>
Mar 10 10:37:59 2kvm2 wdmd[1732]: test failed rem 55 now 140531
ping 140516 close 140526 renewal 140451 expire 140531 client
1673 sanlock_1603cd90-92ef-4c03-922c-cecb282fd00e:1<br>
Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100
140531 [1673]: s3 check_our_lease failed 80<br>
Mar 10 10:38:00 2kvm2 sanlock[1673]: 2017-03-10 10:38:00+0100
140531 [1673]: s3 all pids clear<br>
Mar 10 10:38:01 2kvm2 wdmd[1732]: /dev/watchdog0 reopen<br>
Mar 10 10:38:10 2kvm2 journal: Cannot start job (query, none)
for domain TEST-LBS_EBSAPP; current job is (query, none) owned
by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>)
for (62s, 0s)<br>
Mar 10 10:38:10 2kvm2 journal: Timed out during operation:
cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)<br>
Mar 10 10:38:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL
error receiving from <yajsonrpc.betterAsyncore.Dispatcher
connected ('::1', 40590, 0, 0) at 0x3acdd88>: unexpected eof<br>
Mar 10 10:38:40 2kvm2 journal: Cannot start job (query, none)
for domain TEST1-LBS_ATRYA; current job is (query, none) owned
by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>)
for (47s, 0s)<br>
Mar 10 10:38:40 2kvm2 journal: Timed out during operation:
cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)<br>
Mar 10 10:38:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL
error receiving from <yajsonrpc.betterAsyncore.Dispatcher
connected ('::1', 40592, 0, 0) at 0x3fd5b90>: unexpected eof<br>
Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none)
for domain TEST-LBS_EBSAPP; current job is (query, none) owned
by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>)
for (122s, 0s)<br>
Mar 10 10:39:10 2kvm2 journal: Timed out during operation:
cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)<br>
Mar 10 10:39:10 2kvm2 journal: Cannot start job (query, none)
for domain TEST1-LBS_ATRYA; current job is (query, none) owned
by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>)
for (77s, 0s)<br>
Mar 10 10:39:10 2kvm2 journal: Timed out during operation:
cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)<br>
Mar 10 10:39:11 2kvm2 journal: vdsm vds.dispatcher ERROR SSL
error receiving from <yajsonrpc.betterAsyncore.Dispatcher
connected ('::1', 40594, 0, 0) at 0x2447290>: unexpected eof<br>
Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100
140615 [2576]: s3 delta_renew write time 140 error -202<br>
Mar 10 10:39:23 2kvm2 sanlock[1673]: 2017-03-10 10:39:23+0100
140615 [2576]: s3 renewal error -202 delta_length 144
last_success 140451<br>
Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none)
for domain TEST-LBS_EBSAPP; current job is (query, none) owned
by (3284 remoteDispatchConnectGetAllDomainStats, 0 <null>)
for (152s, 0s)<br>
Mar 10 10:39:40 2kvm2 journal: Timed out during operation:
cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)<br>
Mar 10 10:39:40 2kvm2 journal: Cannot start job (query, none)
for domain TEST1-LBS_ATRYA; current job is (query, none) owned
by (3288 remoteDispatchConnectGetAllDomainStats, 0 <null>)
for (107s, 0s)<br>
Mar 10 10:39:40 2kvm2 journal: Timed out during operation:
cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)<br>
Mar 10 10:39:41 2kvm2 journal: vdsm vds.dispatcher ERROR SSL
error receiving from <yajsonrpc.betterAsyncore.Dispatcher
connected ('::1', 40596, 0, 0) at 0x2472ef0>: unexpected eof<br>
Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42107 blocked
for more than 120 seconds.<br>
Mar 10 10:39:49 2kvm2 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
Mar 10 10:39:49 2kvm2 kernel: qemu-img D
ffff88010dad3e30 0 42107 3631 0x00000080<br>
Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3b30 0000000000000082
ffff8814491f4e70 ffff88010dad3fd8<br>
Mar 10 10:39:49 2kvm2 kernel: ffff88010dad3fd8 ffff88010dad3fd8
ffff8814491f4e70 ffff88301f096c40<br>
Mar 10 10:39:49 2kvm2 kernel: 0000000000000000 7fffffffffffffff
ffff88181f186c00 ffff88010dad3e30<br>
Mar 10 10:39:49 2kvm2 kernel: Call Trace:<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168bbb9>]
schedule+0x29/0x70<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81689609>]
schedule_timeout+0x239/0x2d0<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b15e>]
io_schedule_timeout+0xae/0x130<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8168b1f8>]
io_schedule+0x18/0x20<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff8124d9e5>]
wait_on_sync_kiocb+0x35/0x80<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36091>]
fuse_direct_IO+0x231/0x380 [fuse]<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a6ddd>] ?
cap_inode_need_killpriv+0x2d/0x40<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff812a8cb6>] ?
security_inode_need_killpriv+0x16/0x20<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81219e3f>] ?
dentry_needs_remove_privs.part.13+0x1f/0x30<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81182a2d>]
generic_file_direct_write+0xcd/0x190<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffffa0a36905>]
fuse_file_aio_write+0x185/0x340 [fuse]<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fdabd>]
do_sync_write+0x8d/0xd0<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811fe32d>]
vfs_write+0xbd/0x1e0<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff811ff002>]
SyS_pwrite64+0x92/0xc0<br>
Mar 10 10:39:49 2kvm2 kernel: [<ffffffff81696b09>]
system_call_fastpath+0x16/0x1b<br>
Mar 10 10:39:49 2kvm2 kernel: INFO: task qemu-img:42111 blocked
for more than 120 seconds.<br>
Mar 10 10:39:49 2kvm2 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
Mar 10 10:39:49 2kvm2 kernel: qemu-img D
ffff8818a76e7e30 0 42111 3632 0x00000080<br>
Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7b30 0000000000000082
ffff88188aaeaf10 ffff8818a76e7fd8<br>
Mar 10 10:39:49 2kvm2 kernel: ffff8818a76e7fd8 ffff8818a76e7fd8
ffff88188aaeaf10 ffff88301f156c40<br>
</font><br>
memory<br>
=======<br>
<br>
# cat /proc/meminfo <br>
<font size="-1">MemTotal: 197983472 kB<br>
MemFree: 834228 kB<br>
MemAvailable: 165541204 kB<br>
Buffers: 45548 kB<br>
Cached: 159596272 kB<br>
SwapCached: 119872 kB<br>
Active: 40803264 kB<br>
Inactive: 148022076 kB<br>
Active(anon): 26594112 kB<br>
Inactive(anon): 2626384 kB<br>
Active(file): 14209152 kB<br>
Inactive(file): 145395692 kB<br>
Unevictable: 50488 kB<br>
Mlocked: 50488 kB<br>
SwapTotal: 4194300 kB<br>
SwapFree: 3612188 kB<br>
Dirty: 624 kB<br>
Writeback: 0 kB<br>
AnonPages: 29185032 kB<br>
Mapped: 85176 kB<br>
Shmem: 25908 kB<br>
Slab: 6203384 kB<br>
SReclaimable: 5857240 kB<br>
SUnreclaim: 346144 kB<br>
KernelStack: 19184 kB<br>
PageTables: 86100 kB<br>
NFS_Unstable: 0 kB<br>
Bounce: 0 kB<br>
WritebackTmp: 0 kB<br>
CommitLimit: 103186036 kB<br>
Committed_AS: 52300288 kB<br>
VmallocTotal: 34359738367 kB<br>
VmallocUsed: 1560580 kB<br>
VmallocChunk: 34257341440 kB<br>
HardwareCorrupted: 0 kB<br>
AnonHugePages: 5566464 kB<br>
HugePages_Total: 0<br>
HugePages_Free: 0<br>
HugePages_Rsvd: 0<br>
HugePages_Surp: 0<br>
Hugepagesize: 2048 kB<br>
DirectMap4k: 431292 kB<br>
DirectMap2M: 19382272 kB<br>
DirectMap1G: 183500800 kB</font><br>
<br>
<br>
can anybody help me with this ??<br>
I've got a small tip about swap problem ( in messages), but not
shure .....<br>
The similar problem occured in older versions in gluster/ovirt
testing ( in huge workload freeez - but not fatal overload )<br>
<br>
regards<br>
Paf1<br>
<br>
<br>
</font>
</body>
</html>
--------------C04C9E032CD3A8F03847AFCD--
7 years, 2 months
- 3
- 4
Extend disk in engine appliance in ovirt 4.1.0?
by gflwqs gflwqs
Hi, I have recently installed a new ovirt environment from scratch.
The version is 4.1.0.
I made a 90GB disk for the hosted engine.
When i had installed it and log into the engine i see that it has only
configured 10GB of those 90GB.
So there is 80GB of unconfigured disk unused.
That is fine as long as it had used lvm but it has not meaning i have to
take the engine down to extend the disk?
My questions is:
1. Why does it not configure all disk?
2. Why is it not using lvm?
3. What should i do to extend the disk?
Regards
Christian
7 years, 2 months
- 2
- 1
HE in bad stauts, will not start following storage issue - HELP
by Ian Neilsen
Hi All
I had a storage issue with my gluster volumes running under ovirt hosted.
I now cannot start the hosted engine manager vm from "hosted-engine
--vm-start".
I've scoured the net to find a way, but can't seem to find anything
concrete.
Running Centos7, ovirt 4.0 and gluster 3.8.9
How do I recover the engine manager. Im at a loss!
Engine Status = score between nodes was 0 for all, now node 1 is reading
3400, but all others are 0
{"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"}
Logs from agent.log
==================
INFO::2017-03-09
19:32:52,600::state_decorators::51::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Global maintenance detected
INFO::2017-03-09
19:32:52,603::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM
INFO::2017-03-09
19:32:54,820::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage
INFO::2017-03-09
19:32:54,821::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
INFO::2017-03-09
19:32:59,194::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
INFO::2017-03-09
19:32:59,211::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain
INFO::2017-03-09
19:32:59,328::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Preparing images
INFO::2017-03-09
19:32:59,328::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
Preparing images
INFO::2017-03-09
19:33:01,748::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Reloading vm.conf from the shared storage domain
INFO::2017-03-09
19:33:01,748::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Trying to get a fresher copy of vm configuration from the OVF_STORE
WARNING::2017-03-09
19:33:04,056::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Unable to find OVF_STORE
ERROR::2017-03-09
19:33:04,058::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
ovirt-ha-agent logs
================
ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable
to get vm.conf from OVF_STORE, falling back to initial vm.conf
vdsm
======
vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof
ovirt-ha-broker
============
ovirt-ha-broker cpu_load_no_engine.EngineHealth ERROR Failed to getVmStats:
'pid'
--
Ian Neilsen
Mobile: 0424 379 762
Linkedin: http://au.linkedin.com/in/ianneilsen
Twitter : ineilsen
7 years, 2 months
- 3
- 4
host-only network
by qinglong.dong＠horebdata.cn
This is a multi-part message in MIME format.
------=_001_NextPart562105536544_=----
Content-Type: text/plain;
charset="ISO-8859-1"
Content-Transfer-Encoding: base64
SGksIGFsbA0KICAgICAgICBJIGhhdmUgbm90aWNlZCB0aGF0IGt2bSBzdXBwb3J0cyBob3N0LW9u
bHkgbmV0d29yayBtb2RlLiBTbyBJIHdhbnQgdG8ga25vdyBob3cgdG8gY3JlYXRlIGEgaG9zdC1v
bmx5IHZpbmMgZm9yIGEgdmlydHVhbCBtYWNoaW5lIGluIG92aXJ0Lg0KICAgICAgICBBbnlvbmUg
Y2FuIGhlbHA/IFRoYW5rcyENCg==
------=_001_NextPart562105536544_=----
Content-Type: text/html;
charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable
<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charse=
t=3DISO-8859-1"><style>body { line-height: 1.5; }body { font-size: 10.5pt;=
font-family: ????; color: rgb(0, 0, 0); line-height: 1.5; }body { font-si=
ze: 10.5pt; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>H=
i, all<div> <span style=3D"font-size: 10.5pt; line-heigh=
t: 1.5; background-color: window;"> I have noticed that kvm s=
upports host-only network mode. So I want to know how to create a host-onl=
y vinc for a virtual machine in ovirt.</span></div><div>  =
;<span style=3D"font-size: 10.5pt; line-height: 1.5; background-color: win=
dow;"> Anyone can help? Thanks!</span></div></body></html>
------=_001_NextPart562105536544_=------
7 years, 2 months
- 2
- 1
Re: [ovirt-users] Users Digest, Vol 66, Issue 86
by algotech
--------------7FAFD472380337AE680E33F3
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
It looks that any memory eager running and swap fails afterwards
will write here who's response of this
On 03/10/2017 01:56 PM, users-request(a)ovirt.org wrote:
> Send Users mailing list submissions to
> users(a)ovirt.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.ovirt.org/mailman/listinfo/users
> or, via email, send a message with subject or body 'help' to
> users-request(a)ovirt.org
>
> You can reach the person managing the list at
> users-owner(a)ovirt.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Users digest..."
>
>
> Today's Topics:
>
> 1. Error on Node upgrade 2 (FERNANDO FREDIANI)
> 2. Re: oVirt VM backup and restore (Gianluca Cecchi)
> 3. Re: VDSM hang (Nir Soffer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 10 Mar 2017 09:37:00 -0300
> From: FERNANDO FREDIANI <fernando.frediani(a)upx.com>
> To: "users(a)ovirt.org" <users(a)ovirt.org>
> Subject: [ovirt-users] Error on Node upgrade 2
> Message-ID: <3a306136-25aa-4533-4267-f1b8be58fd87(a)upx.com>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> I am not sure if another email I sent went through but has anyone got
> problems when upgrading a running oVirt-node-ng from 4.1.0 to 4.1.1.
>
> Is the only solution a complete reinstall of the node ?
>
> Thanks
>
> Fernando
>
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 10 Mar 2017 13:52:35 +0100
> From: Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
> To: Juan Hern?ndez <jhernand(a)redhat.com>
> Cc: "users(a)ovirt.org" <users(a)ovirt.org>
> Subject: Re: [ovirt-users] oVirt VM backup and restore
> Message-ID:
> <CAG2kNCwA3-jubtjcz=bzwX+V5-a5V85=HKi5pr=447Ad-tRD4w(a)mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Thu, Mar 9, 2017 at 11:23 AM, Juan Hern?ndez <jhernand(a)redhat.com> wrote:
>
>>> Very good point Gialuca, you are right, the 'persist_memorystate'
>> flag
>>> is 'true' by default, and that makes the pause longer. Would you be
>> so
>>> kind to add it to the 'vm_backup.py' example that is part of version
>> 4
>>> of the SDK?
>>>
>>>
>>> https://github.com/oVirt/ovirt-engine-sdk/blob/master/
>> sdk/examples/vm_backup.py#L143-L151
>>> <https://github.com/oVirt/ovirt-engine-sdk/blob/master/
>> sdk/examples/vm_backup.py#L143-L151>
>>> (Note that that Gibhub is just a mirror, the change needs to be
>> submited
>>> using gerrit.ovirt.org <http://gerrit.ovirt.org>).
>>>
>>>
>>>
>>> I already verified (on a 4.1 infra) that changing vm_backup.py
>>> downloaded yesterday from master this way (apart connection paramters):
>>>
>>> $ diff vm_backup.py vm_backup.py.orig
>>> 150d149
>>> < persist_memorystate=False,
>>>
>>> I get the backup result and snapshot is correctly without memory saved
>>> (and no pause at VM side)
>>>
>> [snip]
>> I see that you already have a gerrit.ovirt.org, so it shouldn't be
>> difficult. The initial setup should be like this:
>>
>> $ git config --global user.name youruser
>> $ git config --global user.email your@email
>> $ git clone ssh://youruser@gerrit.ovirt.org:29418/ovirt-engine-sdk
>> $ gitdir=$(git rev-parse --git-dir); scp -p -P 29418
>> youruser@gerrit.ovirt.org:hooks/commit-msg ${gitdir}/hooks/
>>
>> Then, to submit the patch:
>>
>> $ cd ovirt-engine-sdk
>> $ Edit the vm_backup.py file, and make your changes.
>> $ git commit -a -s
>> $ git push origin HEAD:refs/for/master
>>
>>
> Ok. I found the time to try and apparently it worked as expected.
> The master (you? jenkins CI? ;-) should see my change...
> Just learnt (a little...) another thing
>
> sys admins often try to put an eye inside devs field but the reverse seldom
> happens ;-)
>
7 years, 2 months
- 1
- 0
PHX outage report 10.03.2017
by Evgheni Dereveanchin
Hi everyone,
Tonight we experienced a hardware fault on one of our PHX storage servers.
The faulty server was used to provide storage for multiple production VMs.
Since automatic failover did not happen they became unavailable.
The outage lasted between 09.03.2017 20:36 UTC and 10.03.2017 09:15 UTC
Unavailable services included all of oVirt's CI infrastructure, mailing
lists
and package repositories. Services in other datecenters such as
gerrit.ovirt.org and www.ovirt.org were not affected.
We brought storage back up and this allowed for VMs to be restarted.
If you see tests that failed or didn't run during this period please
re-trigger them.
If there are still persisting issues please report them to the tracker
ticket
that has a more detailed root cause analysis:
https://ovirt-jira.atlassian.net/browse/OVIRT-1244
Sorry for the inconvenience caused. We are working on improving reliability
of the environment to avoid similar incidents from happening in the future.
--
Regards,
Evgheni Dereveanchin
7 years, 2 months
- 1
- 0
oVirt VM backup and restore
by raphael awadallah
Dears Engineers,
I hope you are doing great,
I wish you can help me with a tutorial that shows how to take a scheduled
backups for virtual machines running on oVirt node 4 and how to restore
them.
Best regards
7 years, 2 months
- 8
- 18
Re: [ovirt-users] Unable to add permissions for LDAP users
by Richard Neuboeck
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--2LvNVti3HDOn1MhIrbe5B2snDhjNOnQPn
Content-Type: multipart/mixed; boundary="cUgn38JlIV4VKxpoXvonH2a8Rp00918Nu";
protected-headers="v1"
From: Richard Neuboeck <hawk(a)tbi.univie.ac.at>
To: users <users(a)ovirt.org>
Message-ID: <ce3a6463-b08d-6af6-8adc-66f236ce0bed(a)tbi.univie.ac.at>
Subject: Re: [ovirt-users] Unable to add permissions for LDAP users
References: <71057a13-ee3d-aba0-b005-9fcbc906f375(a)tbi.univie.ac.at>
In-Reply-To: <71057a13-ee3d-aba0-b005-9fcbc906f375(a)tbi.univie.ac.at>
--cUgn38JlIV4VKxpoXvonH2a8Rp00918Nu
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 03/10/2017 09:46 AM, Ondra Machacek wrote:
> So what's your provider 389ds or FreeIPA?
>=20
> Note that both use differrent unique ID. IPA is using 'ipaUniqueID',
> and 389ds is using 'nsuniqueid'. DId you tried both?
Thanks for pointing that out! It works perfectly if I use IPA.
I didn't know they have different identifiers (though it might have
been obvious to me since there is a separate IPA option...). I clung
to the thought that FreeIPA uses 389ds internally.
Thanks a lot!
Richard
>=20
> I can successfully run a search and also login
> from the setup script.
>=20
> After running the setup I rebootet the Engine VM to make sure
> everything is restarted.
>=20
> In the web UI configuration for 'System Permissions' I'm able to
> find users from LDAP but when I try to 'Add' a selected user the UI=
> shows me this error: 'User admin@internal-authz failed to grant
> permission for Role SuperUser on System to User/Group <UNKNOWN>.'.
>=20
> In then engine.log the following lines are generated:
> 2017-03-09 14:02:49,308+01 INFO
> [org.ovirt.engine.core.bll.AddSystemPermissionCommand]
> (org.ovirt.thread.pool-6-thread-4)
> [1ebae5e0-e5f6-49ba-ac80-95266c582893] Running command:
> AddSystemPermissionCommand internal: false. Entities affected : ID=
:
> aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group
> MANIPULATE_PERMISSIONS with role type USER, ID:
> aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group
> ADD_USERS_AND_GROUPS_FROM_DIRECTORY with role type USER
> 2017-03-09 14:02:49,319+01 ERROR
> [org.ovirt.engine.core.bll.AddSystemPermissionCommand]
> (org.ovirt.thread.pool-6-thread-4)
> [1ebae5e0-e5f6-49ba-ac80-95266c582893] Transaction rolled-back for
> command 'org.ovirt.engine.core.bll.AddSystemPermissionCommand'.
> 2017-03-09 14:02:49,328+01 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirect=
or]
> (org.ovirt.thread.pool-6-thread-4)
> [1ebae5e0-e5f6-49ba-ac80-95266c582893] EVENT_ID:
> USER_ADD_SYSTEM_PERMISSION_FAILED(867), Correlation ID:
> 1ebae5e0-e5f6-49ba-ac80-95266c582893, Call Stack: null, Custom Even=
t
> ID: -1, Message: User admin@internal-authz failed to grant
> permission for Role SuperUser on System to User/Group <UNKNOWN>.
>=20
>=20
> So far I've re-run the ldap-setup routine. I made sure all newly
> generated files in /etc/ovirt-engine/[aaa|extensions.d] are owned b=
y
> ovirt:ovirt (instead of root) and have 0600 as permission (instead
> of 0644). That didn't change anything.
>=20
> I've also found an older bug report but for oVirt 3.5
> https://bugzilla.redhat.com/show_bug.cgi?id=3D1121954
> <https://bugzilla.redhat.com/show_bug.cgi?id=3D1121954>
> That didn't reveal any new either.
>=20
> Any ideas what I could try next?
>=20
> Thanks!
> Cheers
> Richard
>=20
>=20
>=20
>=20
> On 10/06/2016 04:36 PM, Ondra Machacek wrote:
> > On 10/06/2016 01:47 PM, Michael Burch wrote:
> >> I'm using the latest ovirt on CentOS7 with the aaa-ldap
> extension.
> >> I can
> >> successfully authenticate as an LDAP user. I can also login as
> >> admin@internal and search for, find, and select LDAP users but I=
> >> cannot
> >> add permissions for them. Each time I get the error "User
> >> admin@internal-authz failed to grant permission for Role
> UserRole on
> >> System to User/Group <UNKNOWN>."
> >
> > This error usually means bad unique attribute used.
> >
> >>
> >>
> >> I have no control over the LDAP server, which uses custom
> >> objectClasses
> >> and uses groupOfNames instead of PosixGroups. I assume I need
> to set
> >> sequence variables to accommodate our group configuration but I'=
m
> >> at a
> >> loss as to where to begin. the The config I have is as follows:
> >>
> >>
> >> include =3D <rfc2307-generic.properties>
> >>
> >> vars.server =3D labauth.lan.lab.org <http://labauth.lan.lab.org>=
> >>
> >> pool.authz.auth.type =3D none
> >> pool.default.serverset.type =3D single
> >> pool.default.serverset.single.server =3D ${global:vars.server}
> >> pool.default.ssl.startTLS =3D true
> >> pool.default.ssl.insecure =3D true
> >>
> >> pool.default.connection-options.connectTimeoutMillis =3D 10000
> >> pool.default.connection-options.responseTimeoutMillis =3D 90000
> >> sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-va=
rs
> >> sequence.my-basedn-init-vars.010.description =3D set baseDN
> >> sequence.my-basedn-init-vars.010.type =3D var-set
> >> sequence.my-basedn-init-vars.010.var-set.variable =3D simple_bas=
eDN
> >> sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB
> >>
> >> sequence-init.init.101-my-objectclass-init-vars =3D
> >> my-objectclass-init-vars
> >> sequence.my-objectclass-init-vars.020.description =3D set
> objectClass
> >> sequence.my-objectclass-init-vars.020.type =3D var-set
> >> sequence.my-objectclass-init-vars.020.var-set.variable =3D
> >> simple_filterUserObject
> >> sequence.my-objectclass-init-vars.020.var-set.value =3D
> >> (objectClass=3DlabPerson)(uid=3D*)
> >>
> >> search.default.search-request.derefPolicy =3D NEVER
> >>
> >> sequence-init.init.900-local-init-vars =3D local-init-vars
> >> sequence.local-init-vars.010.description =3D override name space=
> >> sequence.local-init-vars.010.type =3D var-set
> >> sequence.local-init-vars.010.var-set.variable =3D
> >> simple_namespaceDefault
> >> sequence.local-init-vars.010.var-set.value =3D *
> >
> > What's this^ for? I think it's unusable.
> >
> >>
> >> sequence.local-init-vars.020.description =3D apply filter to use=
rs
> >> sequence.local-init-vars.020.type =3D var-set
> >> sequence.local-init-vars.020.var-set.variable =3D
> >> simple_filterUserObject
> >> sequence.local-init-vars.020.var-set.value =3D
> >> ${seq:simple_filterUserObject}(employeeStatus=3D3)
> >>
> >> sequence.local-init-vars.030.description =3D apply filter to gro=
ups
> >> sequence.local-init-vars.030.type =3D var-set
> >> sequence.local-init-vars.030.var-set.variable =3D
> >> simple_filterGroupObject
> >> sequence.local-init-vars.030.var-set.value =3D
> >> (objectClass=3DgroupOfUniqueNames)
> >
> > This looks as hard to maintain file. I would suggest you to inser=
t
> > into this file just following:
> >
> > include =3D <rfc2307-mycustom.properties>
> >
> > vars.server =3D labauth.lan.lab.org <http://labauth.lan.lab.org>=
> >
> > pool.authz.auth.type =3D none
> > pool.default.serverset.type =3D single
> > pool.default.serverset.single.server =3D ${global:vars.server}
> > pool.default.ssl.startTLS =3D true
> > pool.default.ssl.insecure =3D true
> >
> > pool.default.connection-options.connectTimeoutMillis =3D 10000
> > pool.default.connection-options.responseTimeoutMillis =3D 90000
> >
> > # Set custom base DN
> > sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-va=
rs
> > sequence.my-basedn-init-vars.010.description =3D set baseDN
> > sequence.my-basedn-init-vars.010.type =3D var-set
> > sequence.my-basedn-init-vars.010.var-set.variable =3D simple_bas=
eDN
> > sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB
> >
> > And then create in directory
> > '/usr/share/ovirt-engine-extension-aaa-ldap/profiles/' file
> > 'rfc2307-mycustom.properties' with content:
> >
> > include =3D <rfc2307.properties>
> >
> > sequence-init.init.100-rfc2307-mycustom-init-vars =3D
> > rfc2307-mycustom-init-vars
> > sequence.rfc2307-mycustom-init-vars.010.description =3D set
> unique attr
> > sequence.rfc2307-mycustom-init-vars.010.type =3D var-set
> > sequence.rfc2307-mycustom-init-vars.010.var-set.variable =3D
> > rfc2307_attrsUniqueId
> > sequence.rfc2307-mycustom-init-vars.010.var-set.value =3D
> FIND_THIS_ONE
> >
> > sequence.rfc2307-mycustom-init-vars.020.type =3D var-set
> > sequence.rfc2307-mycustom-init-vars.020.var-set.variable =3D
> > simple_filterUserObject
> > sequence.rfc2307-mycustom-init-vars.020.var-set.value =3D
> >
> (objectClass=3DlabPerson)(employeeStatus=3D3)(${seq:simple_attrsUse=
rName}=3D*)
> >
> >
> >
> > The FIND_*THIS_ONE* replace with the unique attribute of
> labPerson(I
> > guess). It can be extended attribute(+,++).
> >
> > $ LDAPTLS_REQCERT=3Dnever ldapsearch -ZZ -x -b 'o=3DLANLAB' -H
> > ldap://labauth.lan.lab.org <http://labauth.lan.lab.org>
> 'objectClass=3DlabPerson'
> >
> > maybe (or even with two +):
> > $ LDAPTLS_REQCERT=3Dnever ldapsearch -ZZ -x -b 'o=3DLANLAB' -H
> > ldap://labauth.lan.lab.org <http://labauth.lan.lab.org>
> 'objectClass=3DlabPerson' +
> >
> > The question is if even your implementation has unique
> attribute, does
> > it?
> >
> > Also may you share what's your LDAP provider? And maybe if you
> share
> > content of some user it would help as well.
> >
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org <mailto:Users@ovirt.org>
> >> http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
> >>
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org <mailto:Users@ovirt.org>
> > http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
>=20
>=20
> --
> /dev/null
>=20
>=20
--=20
/dev/null
--cUgn38JlIV4VKxpoXvonH2a8Rp00918Nu--
--2LvNVti3HDOn1MhIrbe5B2snDhjNOnQPn
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCgAGBQJYwnegAAoJEA7XCanqEVqIODIQAM4IumlowJYvAoH/+vVTaUT1
pxzYVsOFYHhs32f1in9aqJ37esa1CXngUcB1gHm1QcBixq1FEH1iVEF1Iy22nyXM
gNoOh4R7meoUzqpdlcqYpdTdo3F3v6KgWDRuz9yxH912MZzheXA9BGZ0qhdbQH3S
a1eo2UHNm6wpyS+6CCgUirM8fbrjdb6aTGTuXNQ1px3GaqFFHkQvlWPlENatSxGw
G3IqVoPHhuDNsFDyF6dG6oF2iTbQdzlvk84x3aOztX/nGnxV1dY00/73qEp+8aXK
0krTp0hjiGI0uRiOHChl0DSUuO9VvlTmmqjxYCN1z2ZYHZkMaK590v6YtXmT42tM
VRIsqt6J5HpyLLiQWmCNPPkXf/s7iuG1zAmO+fi4vd5x6gQlW7ao6lLFFRqB8dVr
FuJyv7DW6PcBNjNiG0zCLbGb431t2MK9AaSxF+qb8KmTluDBqQ2trdijM43J+3Jk
/ptoXfpSzWmX2HkDLuaJL0NeQ6F5vqGUZRgu9IkXS2RAvQwLXflm/cvMKsj32EkZ
39gF5k1ViYAdtk9YUwBObor6uxnb0UpdrvZyA5dUEHsYaP6Yw+hVNqi0tloTN9wL
Q+J6CdHK+lRkRppddoEdSmnAtBqhN5/woZEqrBZ1E1jFRUw7KTefbaeUpDbpWQvS
0csSL7D0IScsBJE2kLSA
=73cu
-----END PGP SIGNATURE-----
--2LvNVti3HDOn1MhIrbe5B2snDhjNOnQPn--
7 years, 2 months
- 1
- 0
Unable to add permissions for LDAP users
by Michael Burch
--_000_SN1PR10MB071807A20FF1DCCB62983C19D5C70SN1PR10MB0718namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
I'm using the latest ovirt on CentOS7 with the aaa-ldap extension. I can su=
ccessfully authenticate as an LDAP user. I can also login as admin@internal=
and search for, find, and select LDAP users but I cannot add permissions f=
or them. Each time I get the error "User admin@internal-authz failed to gra=
nt permission for Role UserRole on System to User/Group <UNKNOWN>."
I have no control over the LDAP server, which uses custom objectClasses and=
uses groupOfNames instead of PosixGroups. I assume I need to set sequence =
variables to accommodate our group configuration but I'm at a loss as to wh=
ere to begin. the The config I have is as follows:
include =3D <rfc2307-generic.properties>
vars.server =3D labauth.lan.lab.org
pool.authz.auth.type =3D none
pool.default.serverset.type =3D single
pool.default.serverset.single.server =3D ${global:vars.server}
pool.default.ssl.startTLS =3D true
pool.default.ssl.insecure =3D true
pool.default.connection-options.connectTimeoutMillis =3D 10000
pool.default.connection-options.responseTimeoutMillis =3D 90000
sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-vars
sequence.my-basedn-init-vars.010.description =3D set baseDN
sequence.my-basedn-init-vars.010.type =3D var-set
sequence.my-basedn-init-vars.010.var-set.variable =3D simple_baseDN
sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB
sequence-init.init.101-my-objectclass-init-vars =3D my-objectclass-init-var=
s
sequence.my-objectclass-init-vars.020.description =3D set objectClass
sequence.my-objectclass-init-vars.020.type =3D var-set
sequence.my-objectclass-init-vars.020.var-set.variable =3D simple_filterUse=
rObject
sequence.my-objectclass-init-vars.020.var-set.value =3D (objectClass=3DlabP=
erson)(uid=3D*)
search.default.search-request.derefPolicy =3D NEVER
sequence-init.init.900-local-init-vars =3D local-init-vars
sequence.local-init-vars.010.description =3D override name space
sequence.local-init-vars.010.type =3D var-set
sequence.local-init-vars.010.var-set.variable =3D simple_namespaceDefault
sequence.local-init-vars.010.var-set.value =3D *
sequence.local-init-vars.020.description =3D apply filter to users
sequence.local-init-vars.020.type =3D var-set
sequence.local-init-vars.020.var-set.variable =3D simple_filterUserObject
sequence.local-init-vars.020.var-set.value =3D ${seq:simple_filterUserObjec=
t}(employeeStatus=3D3)
sequence.local-init-vars.030.description =3D apply filter to groups
sequence.local-init-vars.030.type =3D var-set
sequence.local-init-vars.030.var-set.variable =3D simple_filterGroupObject
sequence.local-init-vars.030.var-set.value =3D (objectClass=3DgroupOfUnique=
Names)
--_000_SN1PR10MB071807A20FF1DCCB62983C19D5C70SN1PR10MB0718namp_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
</head>
<body dir=3D"ltr">
<div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;">
<p>I'm using the latest ovirt on CentOS7 with the aaa-ldap extension. I can=
successfully authenticate as an LDAP user. I can also login as admin@inter=
nal and search for, find, and select LDAP users but I cannot add permission=
s for them. Each time I get the
error "<span>User admin@internal-authz failed to grant permission for=
Role UserRole on System to User/Group <UNKNOWN>.</span>"</p>
<p><br>
</p>
<p>I have no control over the LDAP server, which uses custom objectClasses =
and uses groupOfNames instead of PosixGroups. I assume I need to set sequen=
ce variables to accommodate our group configuration but I'm at a loss as to=
where to begin. the The config
I have is as follows:</p>
<p><br>
</p>
<p></p>
<div>include =3D <rfc2307-generic.properties><br>
<br>
vars.server =3D labauth.lan.lab.org<br>
<br>
pool.authz.auth.type =3D none<br>
pool.default.serverset.type =3D single<br>
pool.default.serverset.single.server =3D ${global:vars.server}<br>
pool.default.ssl.startTLS =3D true<br>
pool.default.ssl.insecure =3D true<br>
<br>
pool.default.connection-options.connectTimeoutMillis =3D 10000<br>
pool.default.connection-options.responseTimeoutMillis =3D 90000<br>
sequence-init.init.100-my-basedn-init-vars =3D my-basedn-init-vars<br>
sequence.my-basedn-init-vars.010.description =3D set baseDN<br>
sequence.my-basedn-init-vars.010.type =3D var-set<br>
sequence.my-basedn-init-vars.010.var-set.variable =3D simple_baseDN<br>
sequence.my-basedn-init-vars.010.var-set.value =3D o=3DLANLAB<br>
<br>
sequence-init.init.101-my-objectclass-init-vars =3D my-objectclass-init-var=
s<br>
sequence.my-objectclass-init-vars.020.description =3D set objectClass<br>
sequence.my-objectclass-init-vars.020.type =3D var-set<br>
sequence.my-objectclass-init-vars.020.var-set.variable =3D simple_filterUse=
rObject<br>
sequence.my-objectclass-init-vars.020.var-set.value =3D (objectClass=3DlabP=
erson)(uid=3D*)<br>
<br>
search.default.search-request.derefPolicy =3D NEVER<br>
<br>
sequence-init.init.900-local-init-vars =3D local-init-vars<br>
sequence.local-init-vars.010.description =3D override name space<br>
sequence.local-init-vars.010.type =3D var-set<br>
sequence.local-init-vars.010.var-set.variable =3D simple_namespaceDefault<b=
r>
sequence.local-init-vars.010.var-set.value =3D *<br>
<br>
sequence.local-init-vars.020.description =3D apply filter to users<br>
sequence.local-init-vars.020.type =3D var-set<br>
sequence.local-init-vars.020.var-set.variable =3D simple_filterUserObject<b=
r>
sequence.local-init-vars.020.var-set.value =3D ${seq:simple_filterUserObjec=
t}(employeeStatus=3D3)<br>
<br>
sequence.local-init-vars.030.description =3D apply filter to groups<br>
sequence.local-init-vars.030.type =3D var-set<br>
sequence.local-init-vars.030.var-set.variable =3D simple_filterGroupObject<=
br>
sequence.local-init-vars.030.var-set.value =3D (objectClass=3DgroupOfUnique=
Names)<br>
<br>
<br>
</div>
<p></p>
</div>
</body>
</html>
--_000_SN1PR10MB071807A20FF1DCCB62983C19D5C70SN1PR10MB0718namp_--
7 years, 2 months
- 3
- 3
HE vm fails to migrate due to host CPU incompatibility even after changing cluster CPU type
by Stefano Stagnaro
Hi guys,
I've started an oVirt 4.1 HE deployment on a Broadwell based server.
Then I added to HE a second, older host based on Nehalem. I've
downgraded the cluster CPU type to Nehalem to accommodate host2 and it
finally reached score 3400. However, when I try to migrate HE vm it
fails with the following error:
2017-03-03 20:19:51,814 ERROR (migsrc/b0d38435) [virt.vm]
(vmId='b0d38435-5774-4ca9-ad24-70b57b5bc25d') unsupported configuration:
guest and host CPU are not compatible: Host CPU does not provide
required features: pclmuldq, fma, pcid, x2apic, movbe, tsc-deadline,
aes, xsave, avx, fsgsbase, bmi1, hle, avx2, smep, bmi2, erms, invpcid,
rtm, rdseed, adx, smap, 3dnowprefetch; try using 'Broadwell-noTSX' CPU
model (migration:265)
I believe the problem is in the HE vm XML where the cpu is still
configured as Broadwell. how can I change this specific setting without
losing the deployment? Please find all the relevant logs at the
following link:
https://www.dropbox.com/sh/njl9aofhdw10ses/AADf2Ql4GKVIKcbgLivbmjC2a
Besides that, I believe this is a wrong behavior because HE should
follow cluster properties (otherwise do not reach score 3400); do you
believe is it worth opening a issue on bugzilla?
Thank you,
Stefano.
7 years, 2 months
- 3
- 3