- Users - oVirt List Archives

14 Jan '20

My cluster appears to be experiencing an SPM problem. I recently placed each host in maintenance to move the ovirt management network to another interface. All was successful and all VMs are currently running. However, I'm not facing an SPM contending loop with data center going in and out of responsive status. I have a 3 server HCI setup and all volumes are active and healed, there are no unsynced entries or split brains. Does anyone know how I could diagnose the SPM issue? engine.log: 2020-01-13 22:24:54,777-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [213adf4f] START, GlusterTasksListVDSCommand(HostName = Orchard0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 349f80a9 2020-01-13 22:24:55,231-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [213adf4f] FINISH, GlusterTasksListVDSCommand, return: [], log id: 349f80a9 2020-01-13 22:24:58,245-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterServersListVDSCommand(HostName = Orchard0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 7b04f110 2020-01-13 22:24:58,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM]'} ' 2020-01-13 22:24:58,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] HostName = Orchard1 2020-01-13 22:24:58,888-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = Orchard1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2020-01-13 22:24:59,034-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand, return: [10.12.0.220/24:CONNECTED, orchard1.grove.silverorange.com:CONNECTED, orchard2.grove.silverorange.com:DISCONNECTED], log id: 7b04f110 2020-01-13 22:24:59,049-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterServersListVDSCommand(HostName = Orchard2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 43f1dd82 2020-01-13 22:24:59,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, ConnectStoragePoolVDSCommand(HostName = Orchard1, ConnectStoragePoolVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', vdsId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}), log id: 2b397b31 2020-01-13 22:24:59,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Executing with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active, 23c22a0f-0482-425e-8ada-730cf8ec0751=active, 390c0320-e843-4ff3-a4bb-a9973058447f=active, fb43d33a-82c8-44cb-8169-090cd0d8f56e=active, d70b171e-7488-4d52-8cad-bbc581dbf16e=active, 1f2e9989-9ab3-43d5-971d-568b8feca918=active} 2020-01-13 22:24:59,850-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand, return: [10.12.0.222/24:CONNECTED, 10.11.0.220:CONNECTED, orchard1.grove.silverorange.com:CONNECTED], log id: 43f1dd82 2020-01-13 22:24:59,852-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterVolumesListVDSCommand(HostName = Orchard0, GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 263be6f8 2020-01-13 22:25:00,019-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2b397b31 2020-01-13 22:25:00,036-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] hostFromVds::selectedVds - 'Orchard1', spmStatus 'Free', storage pool 'Default', storage pool version '4.3' 2020-01-13 22:25:00,056-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] starting spm on vds 'Orchard1', storage pool 'Default', prevId '-1', LVER '-1' 2020-01-13 22:25:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, SpmStartVDSCommand(HostName = Orchard1, SpmStartVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', prevId='-1', prevLVER='-1', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'}), log id: 3dea111a 2020-01-13 22:25:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling started: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' 2020-01-13 22:25:00,500-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterVolumesListVDSCommand, return: {3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@743f4102, 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@cad5f8f4, 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f9f55499, c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7a2c16da, 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@33e1ab37, 095fd8fc-5322-4741-8805-fc0bb64b554f=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b28195c0}, log id: 263be6f8 2020-01-13 22:25:03,089-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Failed in 'HSMGetTaskStatusVDS' method 2020-01-13 22:25:03,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling ended: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' task status 'finished' 2020-01-13 22:25:03,090-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Start SPM Task failed - result: 'cleanSuccess', message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',), code = 100 2020-01-13 22:25:03,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling ended, spm status: Free 2020-01-13 22:25:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, HSMClearTaskVDSCommand(HostName = Orchard1, HSMTaskGuidBaseVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', taskId='671d5904-e062-4d45-9eb4-83a6f13657fe'}), log id: 5335606f 2020-01-13 22:25:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 5335606f 2020-01-13 22:25:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, SpmStartVDSCommand, return: SpmStatusResult:{SPM Id='-1', SPM LVER='-1', SPM Status='Free'}, log id: 3dea111a 2020-01-13 22:25:03,117-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: a45e442e-9989-11e8-b0e4-00163e4bf18a Type: StoragePool 2020-01-13 22:25:03,179-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: SpmStart failed 2020-01-13 22:25:03,205-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Irs placed on server 'fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d' failed. Proceed Failover 2020-01-13 22:25:03,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] START, ConnectStoragePoolVDSCommand(HostName = Orchard2, ConnectStoragePoolVDSCommandParameters:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237', vdsId='fd0752d8-2d41-45b0-887a-0ffacbb8a237', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}), log id: 6a7302e9 2020-01-13 22:25:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Executing with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active, 23c22a0f-0482-425e-8ada-730cf8ec0751=active, 390c0320-e843-4ff3-a4bb-a9973058447f=active, fb43d33a-82c8-44cb-8169-090cd0d8f56e=active, d70b171e-7488-4d52-8cad-bbc581dbf16e=active, 1f2e9989-9ab3-43d5-971d-568b8feca918=active} ----------------------------- vdsm.log from one of my hosts: 2020-01-13 22:26:12,434-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,435-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 88772af7-8cf5-433e-8be3-8d0adf0bbf04 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/88772af7-8cf5-433e-8be3-8d0adf0bbf04/88772af7-8cf5-433e-8be3-8d0adf0bbf04.job.0',) 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: a9b11e33-9b93-46a0-a36e-85063fd53ebe (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe/a9b11e33-9b93-46a0-a36e-85063fd53ebe.recover.0',) 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 650f2df4-6489-47e2-af5d-db86a22f01c0 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/650f2df4-6489-47e2-af5d-db86a22f01c0/650f2df4-6489-47e2-af5d-db86a22f01c0.recover.0',) 2020-01-13 22:26:12,487-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,488-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e.job.0',) 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737, in _load self._loadTaskMetaFile(taskDir) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688, in _loadTaskMetaFile self._loadMetaFile(taskFile, self, Task.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743.task',) 2020-01-13 22:26:12,505-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,506-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 75318d21-45b2-4dbd-985c-a7851a10a463 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/75318d21-45b2-4dbd-985c-a7851a10a463/75318d21-45b2-4dbd-985c-a7851a10a463.job.0',) 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 1779b352-022c-49a3-9388-2f688d33cdab (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/1779b352-022c-49a3-9388-2f688d33cdab/1779b352-022c-49a3-9388-2f688d33cdab.job.0',) 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 0302036a-7d99-4685-befb-6fee1602feaf (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/0302036a-7d99-4685-befb-6fee1602feaf/0302036a-7d99-4685-befb-6fee1602feaf.job.0',) 2020-01-13 22:26:12,545-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,546-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 24a825b9-d48d-4134-8aa1-b4db7a9c6ab1 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737, in _load self._loadTaskMetaFile(taskDir) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688, in _loadTaskMetaFile self._loadMetaFile(taskFile, self, Task.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.backup/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.task',) 2020-01-13 22:26:12,560-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,561-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 90d88529-a051-4acd-bef2-d0aa034c15de (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/90d88529-a051-4acd-bef2-d0aa034c15de/90d88529-a051-4acd-bef2-d0aa034c15de.recover.0',) 2020-01-13 22:26:12,607-0400 ERROR (tasks/5) [storage.StoragePool] Unexpected error (sp:383) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in startSpm self.taskMng.recoverDumpedTasks() File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 227, in recoverDumpedTasks self.queueRecovery(task) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 48, in queueRecovery return self._queueTask(task, task.recover) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 54, in _queueTask 'Task id already in use: {0}'.format(task.id)) AddTaskError: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) 2020-01-13 22:26:12,608-0400 ERROR (tasks/5) [storage.StoragePool] failed: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) (sp:384) 2020-01-13 22:26:12,635-0400 ERROR (tasks/5) [storage.TaskManager.Task] (Task='3c7de2a0-597c-4ebe-b4de-689dba26045b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in startSpm self.taskMng.recoverDumpedTasks() File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 227, in recoverDumpedTasks self.queueRecovery(task) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 48, in queueRecovery return self._queueTask(task, task.recover) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 54, in _queueTask 'Task id already in use: {0}'.format(task.id)) AddTaskError: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)

3 3

[ANN] oVirt 4.3.8 Third Release Candidate is now available for testing
by Lev Veyde 14 Jan '20

14 Jan '20

The oVirt Project is pleased to announce the availability of the oVirt 4.3.8 Third Release Candidate for testing, as of January 8th, 2020. This update is a release candidate of the eighth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available Additional Resources: * Read more about the oVirt 4.3.8 release highlights: http://www.ovirt.org/release/4.3.8/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.8/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

4 5

Re: gluster shards not healing
by Strahil 14 Jan '20

14 Jan '20

If you observe similar problems in the future, you can just rsync the files from one brick to the other and run a heal (usually a full heal resolves it all). Most probably the Hypervisor has accesed the file that consists the shards ans the FUSE client noticed that the first host needs healing and uploaded the files . I have used the following to also trigger a heal (not good for large volumes) find /fuse-mountpoint -iname '*' -exec stat {} \; Best Regards, Strahil Nikolov On Jan 14, 2020 02:33, Jayme <jaymef(a)gmail.com> wrote: > > I'm not exactly sure how but it looks like the problem worked itself out after a few hours > > On Mon, Jan 13, 2020 at 5:02 PM Jayme <jaymef(a)gmail.com> wrote: >> >> I have a 3-way replica HCI setup. I recently placed one host in maintenance to perform work on it. When I re-activated it I've noticed that many of my gluster volumes are not completing the heal process. >> >> heal info shows shard files in heal pending. I looked up the files and it appears that they exist on the other two hosts (the ones that remained active) but do not exist on the host that was in maintenance. >> >> I tried to run a manual heal on one of the volumes and then a full heal as well but there are still unhealed shards. The shard files also still do not exist on the maintenance host. Here is an example from one of my volumes: >> >> # gluster volume heal prod_a info >> Brick gluster0:/gluster_bricks/prod_a/prod_a >> Status: Connected >> Number of entries: 0 >> >> Brick gluster1:/gluster_bricks/prod_a/prod_a >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 >> Status: Connected >> Number of entries: 2 >> >> Brick gluster2:/gluster_bricks/prod_a/prod_a >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 >> Status: Connected >> Number of entries: 2 >> >> >> host0: >> >> # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> ls: cannot access /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177: No such file or directory >> >> host1: >> >> # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> >> host2: >> >> # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> >> >> How can I heal these volumes? >> >> Thanks!

1 0

Re: Can we please migrate to 2020 and get a user friendly issues/support tool?
by Strahil 14 Jan '20

14 Jan '20

Last time I used it, I found that at least half of the replies missing... On Jan 13, 2020 18:27, markus.falb(a)mafalb.at wrote: > > there is a web frontend for the mailing list at > https://lists.ovirt.org/archives/list/users@ovirt.org/ > > best regards, markus > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KVXIR6G4D6R77…

1 0

gluster shards not healing
by Jayme 14 Jan '20

14 Jan '20

I have a 3-way replica HCI setup. I recently placed one host in maintenance to perform work on it. When I re-activated it I've noticed that many of my gluster volumes are not completing the heal process. heal info shows shard files in heal pending. I looked up the files and it appears that they exist on the other two hosts (the ones that remained active) but do not exist on the host that was in maintenance. I tried to run a manual heal on one of the volumes and then a full heal as well but there are still unhealed shards. The shard files also still do not exist on the maintenance host. Here is an example from one of my volumes: # gluster volume heal prod_a info Brick gluster0:/gluster_bricks/prod_a/prod_a Status: Connected Number of entries: 0 Brick gluster1:/gluster_bricks/prod_a/prod_a /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 Status: Connected Number of entries: 2 Brick gluster2:/gluster_bricks/prod_a/prod_a /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 Status: Connected Number of entries: 2 host0: # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 ls: cannot access /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177: No such file or directory host1: # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 host2: # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 How can I heal these volumes? Thanks!

1 1

How can I link the cloned disks from a snapshot to the original disks?
by FMGarcia 13 Jan '20

13 Jan '20

Hello, Sorry, I posted in bugzilla, and you advise me than my question isn't error. https://bugzilla.redhat.com/show_bug.cgi?id=1789534 I will attach the same archives and the same text. Description of problem: This problem is bizarre/unusual/convoluted, such that I think that it won't affect to anybody. But the system allows it. In a backup from snapshot with several disks, we can know the original disks from each disk through their names and their sizes. But, and if several disks had the same name and same size? How could we do it without inspect the content of each disk? With the java Sdk, the order of the discs from the original machine to the cloned machine changes, making this path not possible either. In attached files, I show the disks[with different name and order of java Sdk] of the original vm[vm1] and the cloned vm[vm1Clone]. And others[vm2] and [vm2Clone] with the same process, but all disks have the same name. The disks have two sizes: 1Gb and 2Gb. Steps to Reproduce: 1. Create a vm, and attach several disks with same name and same size. 2. Create a backup from snapshot. 3. Now, try to differentiate the disks cloned from original, without looking at their contents. Actual results: I think that it is impossible without I look at their contents. Expected results: Knowing which disk of the cloned machine links to which disk of the original machine. Additional info: If you need more info, request me it Vm.log -> Vm original with different disk name VmClone.log -> Vm clone with different disk name Vm2.log -> Vm original with same disk name Vm2Clone.log -> Cloned vm with same disk name Thanks for your time, Fran

1 0

Re: Please Help, Ovirt Node Hosted Engine Deployment Problems 4.3.2
by William Smyth 13 Jan '20

13 Jan '20

I'm running into the exact same issue as described in this thread. The HostedEngine VM (HE) loses connectivity during the deployment while on the NAT'd network and the deployment fails. My HE VM is never migrated to shared storage because that part of the process hasn't been reached, nor have I been asked for the shared storage details, which are simple NFS shares. I've tried the deployment from the CLI and through Cockpit, both times using the Ansible deployment, and both times reaching the same conclusion. I did manage to connect to the Centos 7 host that is running the HE guest with virt-manager and moved the NIC to the 'ovirtmgmt' vswitch using macvtap, which allowed me to access HE from the outside world. One of the big issues though is that the HE isn't on shared storage and I'm not sure how to move it there. I'm running 2 Centos 7 hosts with different CPU makes, AMD and Intel. So i have two separate single-host clusters but I'm trying to at least have the HA setup for the HE, which I think is still possible. I know that I won't be able to do a live migration, but that's not a big deal. I'm afraid my failed deployment of HE is preventing this and I'm at an impasse. I don't want to give up on oVirt but it's becoming difficult to get my lab environment operational and I'm wasting a lot of time troubleshooting. Any help or suggestions would be greatly appreciated. William Smyth

1 0

Can we please migrate to 2020 and get a user friendly issues/support tool?
by m.skrzetuski＠gmail.com 13 Jan '20

13 Jan '20

Hello everyone, I can speak only for myself but I find mailing lists so 1990. Could we migrate this list to something (user friendly) Jira like? I have to admit it's rather difficult to search an read the archive for old issues in the e-mail format. Kind regards Skrzetuski

3 3

Setting up cockpit?
by m.skrzetuski＠gmail.com 13 Jan '20

13 Jan '20

Hello everyone, I'd like to get cockpit to work because currently when I click "Host Console" on a host I just get "connection refused". I checked and after the engine installation the cockpit service was not running. When I start it, it runs and answers on port 9090, however the SSL certificate is broken. - How do I auto enable cockpit on installation? - How do I supply my own SSL certification to cockpit? Kind regards Skrzetuski

2 8

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

Can you check for any differences betwewn the newly created VM and the old one. Set an alias as follows: alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' Then: virsh dumpxml oldVM virsh dumpxml newVM Maybe somethhing can give a clue what is going on. Best Regards, Strahil NikolovOn Jan 12, 2020 20:03, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Dear Strahil, > > > > Last strange discovery. Newly installed VM – same OS, same NIC’s – connected to the same VLAN’s. Both machines are running on the same virtualization host. > > Newly installed VM – no network issues at all. It works flawlessly. > > > > It is really very strange. > > > > Best, > > Latcho > >

1 0

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

Maybe it's related to the CentOS 5.2 e1000 kernel module . Can you install another CentOS 5.2 and verify that the issue exists? Any chance to upgrade the VM to 5.11 or just update the kernel so it is at least 2.6.25 , so you can try the virtio NIC ? Best Regards, Strahil NikolovOn Jan 12, 2020 20:03, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Dear Strahil, > > > > Last strange discovery. Newly installed VM – same OS, same NIC’s – connected to the same VLAN’s. Both machines are running on the same virtualization host. > > Newly installed VM – no network issues at all. It works flawlessly. > > > > It is really very strange. > > > > Best, > > Latcho > >

1 0

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

I guess I didn't clarify myself. I asked if you can shutdown the VM and then set only this VM's 'Custom compatibility mode' to 4.2 and power up to test. Yet, if the cluster was at 4.2 when you imported , then I am afraid that the 'Custom Compatibility mode' won't make a difference. I'd recommend you to open a bug on bugzilla.redhat.com and share the link with the mailing list. Many of the dev take a look here. But st Regards, Strahil NikolovOn Jan 12, 2020 15:02, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Yes! It is. > > > > After upgrade 4.2 to 4.3 datacenter and cluster compatibility mode was 4.2 when I discovered the issue. Then I set Cluster mode to 4.3 – no luck. It is not so easy to return it to 4.2. > > > > Thank you! > > Best, > > Latcho > > > > > > From:

2 2

Low disk space on Storage
by suporte＠logicworks.pt 12 Jan '20

12 Jan '20

Hi, I'm running ovirt Version:4.3.4.3-1.el7 My filesystem disk has 30 GB free space. Cannot start a VM due to an I/O error storage. When tryng to move the disk to another storage domain get this error: Error while executing action: Cannot move Virtual Disk. Low disk space on Storage Domain DATA4. The sum of pre-allocated disk is the total of the storage domain disk. Any idea what can I do to move a disk to other storage domain? Many thanks -- Jose Ferradeira http://www.logicworks.pt

5 5

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

This seems quite odd. What happens when you set custom compatibility mode to 4.2 ? I think it was in Edit -> system -> Advanced options -> Custom compatibility mode -> 4.2 Best Regards, Strahil NikolovOn Jan 12, 2020 13:24, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Hi Strahil, > > > > Unfortunately virtio drives cannot be used. Guest OS is CentOS 5.2, kernel 2.6.18-92.1.17.el5.centos.plusPAE > > > > The network connectivity of the VM is very short (please see below). Till I connect the VM (via spice) network connectivity is already gone. > > tcpdump –I eth0 –v > > tcpdump: listening on eth0, link-type EN10MB ( Ethernet ), capture 96 bytes > > > > 0 packets captured > > 0 packets received by filer > > 0 packets dropped by kernel > > > > From 172.17.17.7 icmp_seq=60 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=61 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=62 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=63 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=64 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=65 Destination Host Unreachable > > 64 bytes from 172.17.16.16: icmp_seq=66 ttl=64 time=2.19 ms > > 64 bytes from 172.17.16.16: icmp_seq=67 ttl=64 time=0.567 ms > > 64 bytes from 172.17.16.16: icmp_seq=68 ttl=64 time=0.339 ms > > 64 bytes from 172.17.16.16: icmp_seq=69 ttl=64 time=0.337 ms > > 64 bytes from 172.17.16.16: icmp_seq=70 ttl=64 time=0.288 ms > > 64 bytes from 172.17.16.16: icmp_seq=71 ttl=64 time=0.305 ms > > 64 bytes from 172.17.16.16: icmp_seq=72 ttl=64 time=0.293 ms > > 64 bytes from 172.17.16.16: icmp_seq=73 ttl=64 time=0.346 ms > > 64 bytes from 172.17.16.16: icmp_seq=74 ttl=64 time=0.359 ms > > 64 bytes from 172.17.16.16: icmp_seq=75 ttl=64 time=0.479 ms > > 64 bytes from 172.17.16.16: icmp_seq=76 ttl=64 time=0.391 ms > > 64 bytes from 172.17.16.16: icmp_seq=77 ttl=64 time=0.302 ms > > 64 bytes from 172.17.16.16: icmp_seq=78 ttl=64 time=0.327 ms > > 64 bytes from 172.17.16.16: icmp_seq=79 ttl=64 time=0.321 ms > > From 172.17.17.7 icmp_seq=125 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=126 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=127 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=128 Destination Host Unreachable > > > > Thank you! > > Best, > > Latcho > > > > > > From: Strahil Nikolov <hunter86_bg(a)yahoo.com> > Sent: Sunday, January 12, 2020 12:52 PM > To: users <users(a)ovirt.org>; Latchezar Filtchev <Latcho(a)aubg.bg> > Subject: Re: [ovirt-users] Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7 > > > > Hi Lacho, > > > > Can you run the virtio drivers ? > > > > They are the most tested in KVM . > > > > Also, can you run a tcpdump (with e1000) and check what is going on when the network disapperead? > > > > Best Regards, > > Strahil Nikolov > > > > В неделя, 12 януари 2020 г., 09:22:52 ч. Гринуич+2, Latchezar Filtchev <latcho(a)aubg.bg> написа: > > > > > > Dear Strahi, > > > > I tried rtl8139. The behavior is the same. > > > > Best, > > Latcho > > > > > > From: Strahil <hunter86_bg(a)yahoo.com> > Sent: Sunday, January 12, 2020 1:19 AM > To: Latchezar Filtchev <Latcho(a)aubg.bg>; users <users(a)ovirt.org> > Subject: Re: [ovirt-users] Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7 > > > > Hi Latcho, > > Most probably it's a bug. > Have you tried virtio and/or rtl-based NIC ? > > As far as I know, CentOS 5 supports Virtio after Kernel >= 2.6.25 . > > Best Regards, > Strahil Nikolov > > On Jan 11, 2020 19:54, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: >> >> Hello Everyone, >> >> >> >> I am aware my guest OS is out of support but I experienced the following: >> >> >> >> oVirt – 4.2.8 (self-hosted engine) >> >> VM – CentOS 5.2; two VNIC’s ( driver used e1000) connected to different VLAN’s; - no issues with network connectivity >> >> >> >> After upgrade to oVirt 4.3.7 >> >> >> >> The same VM starts normally. Network is available for several seconds (10 – 20 pings) and then it disappears. The machine works but no ping to/from VM. When I am returning the same machine (via export domain) to oVirt 4.2.8 environment – it works as expected. >> >> >> >> Can someone advise on this? Any help will be greatly appreciated. >> >> >> >> Thank you. >> >> Best, >> >> Latcho >> >> >> >> >> >> > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DAPMFAA3XZOUJ…

2 1

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

Hi Latcho, Most probably it's a bug. Have you tried virtio and/or rtl-based NIC ? As far as I know, CentOS 5 supports Virtio after Kernel >= 2.6.25 . Best Regards, Strahil NikolovOn Jan 11, 2020 19:54, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Hello Everyone, > > > > I am aware my guest OS is out of support but I experienced the following: > > > > oVirt – 4.2.8 (self-hosted engine) > > VM – CentOS 5.2; two VNIC’s ( driver used e1000) connected to different VLAN’s; - no issues with network connectivity > > > > After upgrade to oVirt 4.3.7 > > > > The same VM starts normally. Network is available for several seconds (10 – 20 pings) and then it disappears. The machine works but no ping to/from VM. When I am returning the same machine (via export domain) to oVirt 4.2.8 environment – it works as expected. > > > > Can someone advise on this? Any help will be greatly appreciated. > > > > Thank you. > > Best, > > Latcho > > > > > >

3 3

Upload ISO to data domain - checksum mismatch on the brick
by Strahil Nikolov 12 Jan '20

12 Jan '20

Hello Community, I'm currently migrating all my ISOs from the ISO domain to the DATA domain but it seems that the checksum (on brick level) mismatches.The problem is that I have managed to upload several ISOs (one of them 4.9GB) without issues. Restarting the ovirt-imageio-proxy.service and later the HostedEngine VM doesn't help. The logs from the imageio proxy indicate no issues. I thought that the gluster could be the problem ,but this is not the case - I tested via copying (using ovirt1) between the 2 mountpoints and the sha512sum matches. Trying to upload via ovirt2 also doesn't help. Any ideas are appreciated. PS: I'm running latest oVirt. Best Regards,Strahil Nikolov

1 0

HCI Disaster Recovery
by Christian Reiss 12 Jan '20

12 Jan '20

Hey folks, - theoretical question, no live data in jeopardy - Let's say a 3-way HCI cluster is up and running, with engine running, all is well. The setup was done via gui, including gluster. Now I would kill a host, poweroff & disk wipe. Simulating a full node failure. The remaining nodes should keep running on (3 copy sync, no arbiter), vms keep running or will be restarted. I would reinstall using the ovirt node installer on the "failed" node. This would net me with a completely empty, no-gluster setup. What is the oVirt way to recover from this point onward? Thanks for your continued support! <3 -Christian. -- with kind regards, mit freundlichen Gruessen, Christian Reiss

2 2

Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Latchezar Filtchev 11 Jan '20

11 Jan '20

Hello Everyone, I am aware my guest OS is out of support but I experienced the following: oVirt - 4.2.8 (self-hosted engine) VM - CentOS 5.2; two VNIC's ( driver used e1000) connected to different VLAN's; - no issues with network connectivity After upgrade to oVirt 4.3.7 The same VM starts normally. Network is available for several seconds (10 - 20 pings) and then it disappears. The machine works but no ping to/from VM. When I am returning the same machine (via export domain) to oVirt 4.2.8 environment - it works as expected. Can someone advise on this? Any help will be greatly appreciated. Thank you. Best, Latcho

1 0

Hosted-engine.metadata header is corrupted
by du_hongyu＠yeah.net 11 Jan '20

11 Jan '20

Hi my Hosted-engine.metadata header is corrupted. I should how to fix it? thanks Regards Hongyu Du

1 0

Re: HCI Disaster Recovery
by Strahil 10 Jan '20

10 Jan '20

It's actually not so easy. The fastest way to recover is just restore from backup. Otherwise the flow should be: 1. Install the new node (new hostname will be easier). 2. Use gluster's replace-brick to change the dead brick with new one. 3. Once ovirt 's integration with gluster detects the change, you will be able to forcefully remove the dead node. 4. Add the newly installed node to the relevant cluster (with or without hosted-engine deployment) 5. Test to move a low-priority VM to the new host. 6. Power up a test VM on the new host to test the functionality. 7. You can make the new node SPM and test snapahots and new vm creation. As you can see , in order to remove a missing node - it should not be part of the Gluster Cluster. Best Regards, Strahil NikolovOn Jan 10, 2020 20:10, Christian Reiss wrote: > > Hey, > > is there really no ovirt native way to restore a single host and bring > it back into the cluster? > > -Chris. > > On 07.01.2020 09:54, Christian Reiss wrote: > > Hey folks, > > > > - theoretical question, no live data in jeopardy - > > > > Let's say a 3-way HCI cluster is up and running, with engine running, > > all is well. The setup was done via gui, including gluster. > > > > Now I would kill a host, poweroff & disk wipe. Simulating a full node > > failure. > > > > The remaining nodes should keep running on (3 copy sync, no arbiter), > > vms keep running or will be restarted. I would reinstall using the ovirt > > node installer on the "failed" node. > > > > This would net me with a completely empty, no-gluster setup. What is the > > oVirt way to recover from this point onward? > > > > Thanks for your continued support! <3 > > -Christian. > > > > -- > Christian Reiss - email(a)christian-reiss.de /"\ ASCII Ribbon > christian(a)reiss.nrw \ / Campaign > X against HTML > XMPP chris(a)alpha-labs.net / \ in eMails > WEB christian-reiss.de, reiss.nrw > > GPG Retrieval http://gpg.christian-reiss.de > GPG ID ABCD43C5, 0x44E29126ABCD43C5 > GPG fingerprint = 9549 F537 2596 86BA 733C A4ED 44E2 9126 ABCD 43C5 > > "It's better to reign in hell than to serve in heaven.", > John Milton, Paradise lost. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GQECSH72L4SMO…

1 0

Hyperconverged setup - storage architecture - scaling
by Leo David 10 Jan '20

10 Jan '20

Hello Everyone, Reading through the document: "Red Hat Hyperconverged Infrastructure for Virtualization 1.5 Automating RHHI for Virtualization deployment" Regarding storage scaling, i see the following statements: *2.7. SCALINGRed Hat Hyperconverged Infrastructure for Virtualization is supported for one node, and for clusters of 3, 6, 9, and 12 nodes.The initial deployment is either 1 or 3 nodes.There are two supported methods of horizontally scaling Red Hat Hyperconverged Infrastructure for Virtualization:* *1 Add new hyperconverged nodes to the cluster, in sets of three, up to the maximum of 12 hyperconverged nodes.* *2 Create new Gluster volumes using new disks on existing hyperconverged nodes.You cannot create a volume that spans more than 3 nodes, or expand an existing volume so that it spans across more than 3 nodes at a time* *2.9.1. Prerequisites for geo-replicationBe aware of the following requirements and limitations when configuring geo-replication:One geo-replicated volume onlyRed Hat Hyperconverged Infrastructure for Virtualization (RHHI for Virtualization) supports only one geo-replicated volume. Red Hat recommends backing up the volume that stores the data of your virtual machines, as this is usually contains the most valuable data.* ------ Also in oVirtEngine UI, when I add a brick to an existing volume i get the following warning: *"Expanding gluster volume in a hyper-converged setup is not recommended as it could lead to degraded performance. To expand storage for cluster, it is advised to add additional gluster volumes." * Those things are raising a couple of questions that maybe for some for you guys are easy to answer, but for me it creates a bit of confusion... I am also referring to RedHat product documentation, because I treat oVirt as production-ready as RHHI is. *1*. Is there any reason for not going to distributed-replicated volumes ( ie: spread one volume across 6,9, or 12 nodes ) ? - ie: is recomanded that in a 9 nodes scenario I should have 3 separated volumes, but how should I deal with the folowing question *2.* If only one geo-replicated volume can be configured, how should I deal with 2nd and 3rd volume replication for disaster recovery *3.* If the limit of hosts per datacenter is 250, then (in theory ) the recomended way in reaching this treshold would be to create 20 separated oVirt logical clusters with 12 nodes per each ( and datacenter managed from one ha-engine ) ? *4.* In present, I have the folowing one 9 nodes cluster , all hosts contributing with 2 disks each to a single replica 3 distributed replicated volume. They where added to the volume in the following order: node1 - disk1 node2 - disk1 ...... node9 - disk1 node1 - disk2 node2 - disk2 ...... node9 - disk2 At the moment, the volume is arbitrated, but I intend to go for full distributed replica 3. Is this a bad setup ? Why ? It oviously brakes the redhat recommended rules... Is there anyone so kind to discuss on these things ? Thank you very much ! Leo -- Best regards, Leo David -- Best regards, Leo David

3 5

Re: Ovirt OVN help needed
by Strahil 10 Jan '20

10 Jan '20

Hi Dominik, Thanks for your reply. On ovirt1 I got the following: [root@ovirt1 openvswitch]# less ovn-controller.log-20191216.gz 2019-12-15T01:49:02.988Z|00032|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2019-12-16T01:18:02.114Z|00033|vlog|INFO|closing log file ovn-controller.log-20191216.gz (END) Same is on the other node: [root@ovirt2 openvswitch]# less ovn-controller.log-20191216.gz 2019-12-15T01:26:03.477Z|00028|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2019-12-16T01:30:01.718Z|00029|vlog|INFO|closing log file ovn-controller.log-20191216.gz (END) The strange thing is that the geneve tunnels are there: [root@ovirt1 ~]# ovs-vsctl show c0e938f1-b5b5-4d5a-9cda-29dae2986f29 Bridge br-int fail_mode: secure Port "ovn-25cc77-0" Interface "ovn-25cc77-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.64"} Port "ovn-566849-0" Interface "ovn-566849-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.41"} Port br-int Interface br-int type: internal Port "vnet2" Interface "vnet2" ovs_version: "2.11.0" [root@ovirt1 ~]# ovs-vsctl list ports ovs-vsctl: unknown table "ports" [root@ovirt1 ~]# ovs-vsctl list port _uuid : fbf40569-925e-4430-a7c5-c78d58979bbc bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [3207c0cb-3000-40f2-a850-83548f76f090]lacp : [] mac : [] name : "vnet2" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 8947f82d-a089-429b-8843-71371314cb52 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [ec6a6688-e5d6-4346-ac47-ece1b8379440]lacp : [] mac : [] name : br-int other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 72d612be-853e-43e9-8f5c-ce66cef0bebe bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="5668499c-7dd0-41ee-bc5d-2e6ee9cd61c3(a)192.168.1.41"} fake_bridge : false interfaces : [a31574fe-515b-420b-859d-7f2ac729638f]lacp : [] mac : [] name : "ovn-566849-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 2043a15f-ec39-4cc3-b875-7be00423dd7a bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="25cc77b3-046f-45c5-af0c-ffb2f77d73f1(a)192.168.1.64"} fake_bridge : false interfaces : [f9a9e3ff-070e-4044-b601-7f7394dc295f]lacp : [] mac : [] name : "ovn-25cc77-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] [root@ovirt1 ~]# [root@ovirt2 ~]# ovs-vsctl show 3dbab138-6b90-44c5-af05-b8a944c9bf20 Bridge br-int fail_mode: secure Port "ovn-baa019-0" Interface "ovn-baa019-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.90"} Port br-int Interface br-int type: internal Port "vnet5" Interface "vnet5" Port "ovn-566849-0" Interface "ovn-566849-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.1.41"} ovs_version: "2.11.0" [root@ovirt2 ~]# ovs-vsctl list port _uuid : 151e1188-f07a-4750-a620-392a08e7e7fe bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="baa0199e-d1a4-484c-af13-a41bcad19dbc(a)192.168.1.90"} fake_bridge : false interfaces : [4d4bc12a-609a-4917-b839-d4f652acdc33]lacp : [] mac : [] name : "ovn-baa019-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : 3a862f96-b3ec-46a9-bcf6-f385e5def410 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [777f2819-ca27-4890-8d2f-11349ca0d398]lacp : [] mac : [] name : br-int other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : a65109fa-f8b4-4670-8ae8-a2bd0bf6aba3 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {ovn-chassis-id="5668499c-7dd0-41ee-bc5d-2e6ee9cd61c3(a)192.168.1.41"} fake_bridge : false interfaces : [ed442077-f897-4e0b-97a1-a8051e9c3d56]lacp : [] mac : [] name : "ovn-566849-0" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] _uuid : a1622e6f-fcd0-4a8a-b259-ca4d0ccf1cd2 bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [ca368654-54f3-49d0-a71c-8894426df6bf]lacp : [] mac : [] name : "vnet5" other_config : {} protected : false qos : [] rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] [root@ovirt2 ~]# Best Regards, Strahil Nikolov On Dec 16, 2019 23:28, Dominik Holler <dholler(a)redhat.com> wrote: > > > > On Sat, Dec 14, 2019 at 11:36 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: >> >> Hi Dominik, >> >> yes I was looking for those settings. >> >> I have added again the external provider , but I guess the mess is even bigger as I made some stupid decisions (like removing 2 port groups :) without knowing what I'm doing) . >> Sadly I can't remove all packages on the engine and hosts and reinstall them from scratch. >> >> Pip fails to install the openstacksdk (centOS7 is not great for such tasks) on the engine and my lack of knowledge in OVN makes it even more difficult. >> >> So the symptoms are that 2 machines can communicate with each other only if they are on the same host ,while on separate - no communications is happening. >> > > This indicates that the tunnels between the hosts are not created. > Can you please check the /var/log/openvswitch/ovn-controller.log on both hosts for errors and warnings, or share parts of the files here? > If this does not point us to a problem, ovn has to be reconfigured. If possible, most easy way to do this would be to ensure that > ovirt-provider-ovn is the default network provider of the cluster of the hosts, put one host after another in maintance mode and reinstall. > > >> >> How I created the network via UI: >> >> 1. Networks - new >> 2. Fill in the name >> 3. Create on external provider >> 4. Network Port security -> disabled (even undefined does not work) >> 5.Connect to physical network -> ovirtmgmt >> >> >> I would be happy to learn more about OVN and thus I would like to make it work. >> >> Here is some info from the engine: >> >> [root@engine ~]# ovn-nbctl show >> switch 1288ed26-471c-4bc2-8a7d-4531f306f44c (ovirt-pxelan-2a88b2e0-d04b-4196-ad50-074501e4ed08) >> port c1eba112-5eed-4c04-b25c-d3dcfb934546 >> addresses: ["56:6f:5a:65:00:06"] >> port 8b52ab60-f474-4d51-b258-cb2e0a53c34a >> type: localnet >> addresses: ["unknown"] >> port b2753040-881b-487a-92a1-9721da749be4 >> addresses: ["56:6f:5a:65:00:09"] >> [root@engine ~]# ovn-sbctl show >> Chassis "5668499c-7dd0-41ee-bc5d-2e6ee9cd61c3" >> hostname: "ovirt3.localdomain" >> Encap geneve >> ip: "192.168.1.41" >> options: {csum="true"} >> Chassis "baa0199e-d1a4-484c-af13-a41bcad19dbc" >> hostname: "ovirt1.localdomain" >> Encap geneve >> ip: "192.168.1.90" >> options: {csum="true"} >> Chassis "25cc77b3-046f-45c5-af0c-ffb2f77d73f1" >> hostname: "ovirt2.localdomain" >> Encap geneve >> ip: "192.168.1.64" >> options: {csum="true"} >> Port_Binding "b2753040-881b-487a-92a1-9721da749be4" >> Port_Binding &quo

4 8

is there any feature of load balancing for engine??
by yam yam 10 Jan '20

10 Jan '20

Hello, given massive oVirt environment, I think single engine looks too small to deal with all workloads. so, I want to make active-active engine cluster for distributing workloads. is it possible for an oVirt environment to be made up of multiple engines & DBs for load balancing?

3 4

Re: is there any feature of load balancing for engine??
by Strahil 10 Jan '20

10 Jan '20

It is not a good idea. If you think that the engine can't cope with the load, add a separate engine for the specific cluster(s). What is the count of Datacenters, Clusters, Hosts, VMs & Storage domains ? Best Regards, Strahil NikolovOn Jan 8, 2020 03:51, yam yam <hyunooudy(a)gmail.com> wrote: > > Hello, > > given massive oVirt environment, I think single engine looks too small to deal with all workloads. > so, I want to make active-active engine cluster for distributing workloads. > is it possible for an oVirt environment to be made up of multiple engines & DBs for load balancing? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XCO5DJS63YKC6…

2 1

Re: Setting up cockpit?
by Strahil 10 Jan '20

10 Jan '20

On Jan 9, 2020 18:58, m.skrzetuski(a)gmail.com wrote: > > Hello everyone, > > I'd like to get cockpit to work because currently when I click "Host Console" on a host I just get "connection refused". I checked and after the engine installation the cockpit service was not running. When I start it, it runs and answers on port 9090, however the SSL certificate is broken. > > - How do I auto enable cockpit on installation? systemctl enable --now cockpit.socket -H < destination server> > - How do I supply my own SSL certification to cockpit? Put them in /etc/cockpit/ws-certs.d/ Source:https://www.redhat.com/en/blog/linux-system-administration-managemen… > > Kind regards > Skrzetuski Best Regards, Strahil Nikolov

1 0