- Users - oVirt List Archives

ovirt-provider-ovn external network provider
by p.staniforth＠leedsbeckett.ac.uk 14 Jan '20

14 Jan '20

Hello I was testing the ovirt-provider-ovn which was working but at some point seems to have stopped. The logical networks are created and VMs have their ports created and attached on the hosts they run or migrate to but their are no geneve tunnels created and on the engine host "ovn-sbctl show" produces no output. Any help appreciated. Thanks, Paul S.

3 2

Ovirt installation with USB
by Nazan CENGİZ 14 Jan '20

14 Jan '20

Hi all, I am trying install ovirt-node-ng-installer-4.3.5-2019080513.el7<https://resources.ovirt.org/pub/ovirt-4.3/iso/ovirt-node-ng-installer/4.3.5…> .I want setup on USB.I used UltraISO and USB Converter vs for converting.But it failed on below; "dracut-initqueue Warning dracut timeout": dracut/: Do you have any suggestions? Thanks. [cid:image56c8fc.PNG@b74488ec.4c807a5f]<http://www.havelsan.com.tr> [cid:image1ba961.JPG@c094eb48.40ac6f8b] Nazan CENGİZ AR-GE MÜHENDİSİ Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE [cid:image3092b8.PNG@a1ffaf9d.43a701c4] +90 312 219 57 87 [cid:image93a440.PNG@3f130358.4c891e67] +90 312 219 57 97 [cid:image7b8ddf.JPG@3fa3e4a7.40a7c68c] YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari> LEGAL NOTICE: This e-mail is subject to the Terms and Conditions document which can be accessed with this link. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari> [http://www.havelsan.com.tr/Library/images/mail/email.jpg] Lütfen gerekmedikçe bu sayfa ve eklerini yazdırmayınız / Please consider the environment before printing this email

3 2

Re: Upload ISO to data domain - checksum mismatch on the brick
by Strahil 14 Jan '20

14 Jan '20

Hello Community, Just a short update - uploading on the next day works without issues. I will check it again ... Maybe too many isos in a row makes it crazy. Note: I have opened an RFE , so we can provide the ISO checksum, so the proxy can verify it: https://bugzilla.redhat.com/show_bug.cgi?id=1787906 Best Regards, Strahil NikolovOn Jan 12, 2020 12:56, Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: > > Hello Community, > > I'm currently migrating all my ISOs from the ISO domain to the DATA domain but it seems that the checksum (on brick level) mismatches. > The problem is that I have managed to upload several ISOs (one of them 4.9GB) without issues. > > Restarting the ovirt-imageio-proxy.service and later the HostedEngine VM doesn't help. > > The logs from the imageio proxy indicate no issues. > > I thought that the gluster could be the problem ,but this is not the case - I tested via copying (using ovirt1) between the 2 mountpoints and the sha512sum matches. > > Trying to upload via ovirt2 also doesn't help. > > Any ideas are appreciated. > > PS: I'm running latest oVirt. > > Best Regards, > Strahil Nikolov

1 0

Re: Ovirt installation with USB
by Strahil 14 Jan '20

14 Jan '20

It's CentOS based -> either use the tools in the CentOS wiki (or anything in the Fedora) or use 'dd' in any Linux/UNIX distribution. Best Regards, Strahil NikolovOn Jan 14, 2020 13:43, Nazan CENGİZ <nazancengiz(a)havelsan.com.tr> wrote: > > Hi all, > > I am trying install ovirt-node-ng-installer-4.3.5-2019080513.el7 .I want setup on USB.I used UltraISO and USB Converter vs for converting.But it failed on below; > > "dracut-initqueue Warning dracut timeout": > > > dracut/: > > > Do you have any suggestions? > > Thanks. > > > > Nazan CENGİZ > AR-GE MÜHENDİSİ > Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE > +90 312 219 57 87 > +90 312 219 57 97 > YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir. > LEGAL NOTICE: <

1 0

SPM contending loop
by Jayme 14 Jan '20

14 Jan '20

My cluster appears to be experiencing an SPM problem. I recently placed each host in maintenance to move the ovirt management network to another interface. All was successful and all VMs are currently running. However, I'm not facing an SPM contending loop with data center going in and out of responsive status. I have a 3 server HCI setup and all volumes are active and healed, there are no unsynced entries or split brains. Does anyone know how I could diagnose the SPM issue? engine.log: 2020-01-13 22:24:54,777-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [213adf4f] START, GlusterTasksListVDSCommand(HostName = Orchard0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 349f80a9 2020-01-13 22:24:55,231-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [213adf4f] FINISH, GlusterTasksListVDSCommand, return: [], log id: 349f80a9 2020-01-13 22:24:58,245-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterServersListVDSCommand(HostName = Orchard0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 7b04f110 2020-01-13 22:24:58,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM]'} ' 2020-01-13 22:24:58,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] HostName = Orchard1 2020-01-13 22:24:58,888-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = Orchard1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2020-01-13 22:24:59,034-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand, return: [10.12.0.220/24:CONNECTED, orchard1.grove.silverorange.com:CONNECTED, orchard2.grove.silverorange.com:DISCONNECTED], log id: 7b04f110 2020-01-13 22:24:59,049-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterServersListVDSCommand(HostName = Orchard2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 43f1dd82 2020-01-13 22:24:59,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, ConnectStoragePoolVDSCommand(HostName = Orchard1, ConnectStoragePoolVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', vdsId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}), log id: 2b397b31 2020-01-13 22:24:59,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Executing with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active, 23c22a0f-0482-425e-8ada-730cf8ec0751=active, 390c0320-e843-4ff3-a4bb-a9973058447f=active, fb43d33a-82c8-44cb-8169-090cd0d8f56e=active, d70b171e-7488-4d52-8cad-bbc581dbf16e=active, 1f2e9989-9ab3-43d5-971d-568b8feca918=active} 2020-01-13 22:24:59,850-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand, return: [10.12.0.222/24:CONNECTED, 10.11.0.220:CONNECTED, orchard1.grove.silverorange.com:CONNECTED], log id: 43f1dd82 2020-01-13 22:24:59,852-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterVolumesListVDSCommand(HostName = Orchard0, GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 263be6f8 2020-01-13 22:25:00,019-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2b397b31 2020-01-13 22:25:00,036-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] hostFromVds::selectedVds - 'Orchard1', spmStatus 'Free', storage pool 'Default', storage pool version '4.3' 2020-01-13 22:25:00,056-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] starting spm on vds 'Orchard1', storage pool 'Default', prevId '-1', LVER '-1' 2020-01-13 22:25:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, SpmStartVDSCommand(HostName = Orchard1, SpmStartVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', prevId='-1', prevLVER='-1', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'}), log id: 3dea111a 2020-01-13 22:25:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling started: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' 2020-01-13 22:25:00,500-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterVolumesListVDSCommand, return: {3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@743f4102, 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@cad5f8f4, 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f9f55499, c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7a2c16da, 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@33e1ab37, 095fd8fc-5322-4741-8805-fc0bb64b554f=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b28195c0}, log id: 263be6f8 2020-01-13 22:25:03,089-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Failed in 'HSMGetTaskStatusVDS' method 2020-01-13 22:25:03,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling ended: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' task status 'finished' 2020-01-13 22:25:03,090-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Start SPM Task failed - result: 'cleanSuccess', message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',), code = 100 2020-01-13 22:25:03,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling ended, spm status: Free 2020-01-13 22:25:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, HSMClearTaskVDSCommand(HostName = Orchard1, HSMTaskGuidBaseVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', taskId='671d5904-e062-4d45-9eb4-83a6f13657fe'}), log id: 5335606f 2020-01-13 22:25:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 5335606f 2020-01-13 22:25:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, SpmStartVDSCommand, return: SpmStatusResult:{SPM Id='-1', SPM LVER='-1', SPM Status='Free'}, log id: 3dea111a 2020-01-13 22:25:03,117-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: a45e442e-9989-11e8-b0e4-00163e4bf18a Type: StoragePool 2020-01-13 22:25:03,179-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: SpmStart failed 2020-01-13 22:25:03,205-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Irs placed on server 'fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d' failed. Proceed Failover 2020-01-13 22:25:03,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] START, ConnectStoragePoolVDSCommand(HostName = Orchard2, ConnectStoragePoolVDSCommandParameters:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237', vdsId='fd0752d8-2d41-45b0-887a-0ffacbb8a237', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}), log id: 6a7302e9 2020-01-13 22:25:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Executing with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active, 23c22a0f-0482-425e-8ada-730cf8ec0751=active, 390c0320-e843-4ff3-a4bb-a9973058447f=active, fb43d33a-82c8-44cb-8169-090cd0d8f56e=active, d70b171e-7488-4d52-8cad-bbc581dbf16e=active, 1f2e9989-9ab3-43d5-971d-568b8feca918=active} ----------------------------- vdsm.log from one of my hosts: 2020-01-13 22:26:12,434-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,435-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 88772af7-8cf5-433e-8be3-8d0adf0bbf04 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/88772af7-8cf5-433e-8be3-8d0adf0bbf04/88772af7-8cf5-433e-8be3-8d0adf0bbf04.job.0',) 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: a9b11e33-9b93-46a0-a36e-85063fd53ebe (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe/a9b11e33-9b93-46a0-a36e-85063fd53ebe.recover.0',) 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 650f2df4-6489-47e2-af5d-db86a22f01c0 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/650f2df4-6489-47e2-af5d-db86a22f01c0/650f2df4-6489-47e2-af5d-db86a22f01c0.recover.0',) 2020-01-13 22:26:12,487-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,488-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e.job.0',) 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737, in _load self._loadTaskMetaFile(taskDir) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688, in _loadTaskMetaFile self._loadMetaFile(taskFile, self, Task.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743.task',) 2020-01-13 22:26:12,505-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,506-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 75318d21-45b2-4dbd-985c-a7851a10a463 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/75318d21-45b2-4dbd-985c-a7851a10a463/75318d21-45b2-4dbd-985c-a7851a10a463.job.0',) 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 1779b352-022c-49a3-9388-2f688d33cdab (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/1779b352-022c-49a3-9388-2f688d33cdab/1779b352-022c-49a3-9388-2f688d33cdab.job.0',) 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 0302036a-7d99-4685-befb-6fee1602feaf (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/0302036a-7d99-4685-befb-6fee1602feaf/0302036a-7d99-4685-befb-6fee1602feaf.job.0',) 2020-01-13 22:26:12,545-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,546-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 24a825b9-d48d-4134-8aa1-b4db7a9c6ab1 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737, in _load self._loadTaskMetaFile(taskDir) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688, in _loadTaskMetaFile self._loadMetaFile(taskFile, self, Task.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.backup/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.task',) 2020-01-13 22:26:12,560-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,561-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 90d88529-a051-4acd-bef2-d0aa034c15de (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/90d88529-a051-4acd-bef2-d0aa034c15de/90d88529-a051-4acd-bef2-d0aa034c15de.recover.0',) 2020-01-13 22:26:12,607-0400 ERROR (tasks/5) [storage.StoragePool] Unexpected error (sp:383) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in startSpm self.taskMng.recoverDumpedTasks() File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 227, in recoverDumpedTasks self.queueRecovery(task) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 48, in queueRecovery return self._queueTask(task, task.recover) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 54, in _queueTask 'Task id already in use: {0}'.format(task.id)) AddTaskError: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) 2020-01-13 22:26:12,608-0400 ERROR (tasks/5) [storage.StoragePool] failed: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) (sp:384) 2020-01-13 22:26:12,635-0400 ERROR (tasks/5) [storage.TaskManager.Task] (Task='3c7de2a0-597c-4ebe-b4de-689dba26045b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in startSpm self.taskMng.recoverDumpedTasks() File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 227, in recoverDumpedTasks self.queueRecovery(task) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 48, in queueRecovery return self._queueTask(task, task.recover) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 54, in _queueTask 'Task id already in use: {0}'.format(task.id)) AddTaskError: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)

3 3

[ANN] oVirt 4.3.8 Third Release Candidate is now available for testing
by Lev Veyde 14 Jan '20

14 Jan '20

The oVirt Project is pleased to announce the availability of the oVirt 4.3.8 Third Release Candidate for testing, as of January 8th, 2020. This update is a release candidate of the eighth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available Additional Resources: * Read more about the oVirt 4.3.8 release highlights: http://www.ovirt.org/release/4.3.8/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.8/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

4 5

Re: gluster shards not healing
by Strahil 14 Jan '20

14 Jan '20

If you observe similar problems in the future, you can just rsync the files from one brick to the other and run a heal (usually a full heal resolves it all). Most probably the Hypervisor has accesed the file that consists the shards ans the FUSE client noticed that the first host needs healing and uploaded the files . I have used the following to also trigger a heal (not good for large volumes) find /fuse-mountpoint -iname '*' -exec stat {} \; Best Regards, Strahil Nikolov On Jan 14, 2020 02:33, Jayme <jaymef(a)gmail.com> wrote: > > I'm not exactly sure how but it looks like the problem worked itself out after a few hours > > On Mon, Jan 13, 2020 at 5:02 PM Jayme <jaymef(a)gmail.com> wrote: >> >> I have a 3-way replica HCI setup. I recently placed one host in maintenance to perform work on it. When I re-activated it I've noticed that many of my gluster volumes are not completing the heal process. >> >> heal info shows shard files in heal pending. I looked up the files and it appears that they exist on the other two hosts (the ones that remained active) but do not exist on the host that was in maintenance. >> >> I tried to run a manual heal on one of the volumes and then a full heal as well but there are still unhealed shards. The shard files also still do not exist on the maintenance host. Here is an example from one of my volumes: >> >> # gluster volume heal prod_a info >> Brick gluster0:/gluster_bricks/prod_a/prod_a >> Status: Connected >> Number of entries: 0 >> >> Brick gluster1:/gluster_bricks/prod_a/prod_a >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 >> Status: Connected >> Number of entries: 2 >> >> Brick gluster2:/gluster_bricks/prod_a/prod_a >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 >> Status: Connected >> Number of entries: 2 >> >> >> host0: >> >> # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> ls: cannot access /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177: No such file or directory >> >> host1: >> >> # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> >> host2: >> >> # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 >> >> >> How can I heal these volumes? >> >> Thanks!

1 0

Re: Can we please migrate to 2020 and get a user friendly issues/support tool?
by Strahil 14 Jan '20

14 Jan '20

Last time I used it, I found that at least half of the replies missing... On Jan 13, 2020 18:27, markus.falb(a)mafalb.at wrote: > > there is a web frontend for the mailing list at > https://lists.ovirt.org/archives/list/users@ovirt.org/ > > best regards, markus > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KVXIR6G4D6R77…

1 0

gluster shards not healing
by Jayme 14 Jan '20

14 Jan '20

I have a 3-way replica HCI setup. I recently placed one host in maintenance to perform work on it. When I re-activated it I've noticed that many of my gluster volumes are not completing the heal process. heal info shows shard files in heal pending. I looked up the files and it appears that they exist on the other two hosts (the ones that remained active) but do not exist on the host that was in maintenance. I tried to run a manual heal on one of the volumes and then a full heal as well but there are still unhealed shards. The shard files also still do not exist on the maintenance host. Here is an example from one of my volumes: # gluster volume heal prod_a info Brick gluster0:/gluster_bricks/prod_a/prod_a Status: Connected Number of entries: 0 Brick gluster1:/gluster_bricks/prod_a/prod_a /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 Status: Connected Number of entries: 2 Brick gluster2:/gluster_bricks/prod_a/prod_a /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 /.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.178 Status: Connected Number of entries: 2 host0: # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 ls: cannot access /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177: No such file or directory host1: # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 host2: # ls -al /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 -rw-rw----. 2 root root 67108864 Jan 13 16:57 /gluster_bricks/prod_a/prod_a/.shard/a746f8d2-5044-4d20-b525-24456e6f6f16.177 How can I heal these volumes? Thanks!

1 1

How can I link the cloned disks from a snapshot to the original disks?
by FMGarcia 13 Jan '20

13 Jan '20

Hello, Sorry, I posted in bugzilla, and you advise me than my question isn't error. https://bugzilla.redhat.com/show_bug.cgi?id=1789534 I will attach the same archives and the same text. Description of problem: This problem is bizarre/unusual/convoluted, such that I think that it won't affect to anybody. But the system allows it. In a backup from snapshot with several disks, we can know the original disks from each disk through their names and their sizes. But, and if several disks had the same name and same size? How could we do it without inspect the content of each disk? With the java Sdk, the order of the discs from the original machine to the cloned machine changes, making this path not possible either. In attached files, I show the disks[with different name and order of java Sdk] of the original vm[vm1] and the cloned vm[vm1Clone]. And others[vm2] and [vm2Clone] with the same process, but all disks have the same name. The disks have two sizes: 1Gb and 2Gb. Steps to Reproduce: 1. Create a vm, and attach several disks with same name and same size. 2. Create a backup from snapshot. 3. Now, try to differentiate the disks cloned from original, without looking at their contents. Actual results: I think that it is impossible without I look at their contents. Expected results: Knowing which disk of the cloned machine links to which disk of the original machine. Additional info: If you need more info, request me it Vm.log -> Vm original with different disk name VmClone.log -> Vm clone with different disk name Vm2.log -> Vm original with same disk name Vm2Clone.log -> Cloned vm with same disk name Thanks for your time, Fran

1 0

Re: Please Help, Ovirt Node Hosted Engine Deployment Problems 4.3.2
by William Smyth 13 Jan '20

13 Jan '20

I'm running into the exact same issue as described in this thread. The HostedEngine VM (HE) loses connectivity during the deployment while on the NAT'd network and the deployment fails. My HE VM is never migrated to shared storage because that part of the process hasn't been reached, nor have I been asked for the shared storage details, which are simple NFS shares. I've tried the deployment from the CLI and through Cockpit, both times using the Ansible deployment, and both times reaching the same conclusion. I did manage to connect to the Centos 7 host that is running the HE guest with virt-manager and moved the NIC to the 'ovirtmgmt' vswitch using macvtap, which allowed me to access HE from the outside world. One of the big issues though is that the HE isn't on shared storage and I'm not sure how to move it there. I'm running 2 Centos 7 hosts with different CPU makes, AMD and Intel. So i have two separate single-host clusters but I'm trying to at least have the HA setup for the HE, which I think is still possible. I know that I won't be able to do a live migration, but that's not a big deal. I'm afraid my failed deployment of HE is preventing this and I'm at an impasse. I don't want to give up on oVirt but it's becoming difficult to get my lab environment operational and I'm wasting a lot of time troubleshooting. Any help or suggestions would be greatly appreciated. William Smyth

1 0

Can we please migrate to 2020 and get a user friendly issues/support tool?
by m.skrzetuski＠gmail.com 13 Jan '20

13 Jan '20

Hello everyone, I can speak only for myself but I find mailing lists so 1990. Could we migrate this list to something (user friendly) Jira like? I have to admit it's rather difficult to search an read the archive for old issues in the e-mail format. Kind regards Skrzetuski

3 3

Setting up cockpit?
by m.skrzetuski＠gmail.com 13 Jan '20

13 Jan '20

Hello everyone, I'd like to get cockpit to work because currently when I click "Host Console" on a host I just get "connection refused". I checked and after the engine installation the cockpit service was not running. When I start it, it runs and answers on port 9090, however the SSL certificate is broken. - How do I auto enable cockpit on installation? - How do I supply my own SSL certification to cockpit? Kind regards Skrzetuski

2 8

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

Can you check for any differences betwewn the newly created VM and the old one. Set an alias as follows: alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' Then: virsh dumpxml oldVM virsh dumpxml newVM Maybe somethhing can give a clue what is going on. Best Regards, Strahil NikolovOn Jan 12, 2020 20:03, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Dear Strahil, > > > > Last strange discovery. Newly installed VM – same OS, same NIC’s – connected to the same VLAN’s. Both machines are running on the same virtualization host. > > Newly installed VM – no network issues at all. It works flawlessly. > > > > It is really very strange. > > > > Best, > > Latcho > >

1 0

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

Maybe it's related to the CentOS 5.2 e1000 kernel module . Can you install another CentOS 5.2 and verify that the issue exists? Any chance to upgrade the VM to 5.11 or just update the kernel so it is at least 2.6.25 , so you can try the virtio NIC ? Best Regards, Strahil NikolovOn Jan 12, 2020 20:03, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Dear Strahil, > > > > Last strange discovery. Newly installed VM – same OS, same NIC’s – connected to the same VLAN’s. Both machines are running on the same virtualization host. > > Newly installed VM – no network issues at all. It works flawlessly. > > > > It is really very strange. > > > > Best, > > Latcho > >

1 0

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

I guess I didn't clarify myself. I asked if you can shutdown the VM and then set only this VM's 'Custom compatibility mode' to 4.2 and power up to test. Yet, if the cluster was at 4.2 when you imported , then I am afraid that the 'Custom Compatibility mode' won't make a difference. I'd recommend you to open a bug on bugzilla.redhat.com and share the link with the mailing list. Many of the dev take a look here. But st Regards, Strahil NikolovOn Jan 12, 2020 15:02, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Yes! It is. > > > > After upgrade 4.2 to 4.3 datacenter and cluster compatibility mode was 4.2 when I discovered the issue. Then I set Cluster mode to 4.3 – no luck. It is not so easy to return it to 4.2. > > > > Thank you! > > Best, > > Latcho > > > > > > From:

2 2

Low disk space on Storage
by suporte＠logicworks.pt 12 Jan '20

12 Jan '20

Hi, I'm running ovirt Version:4.3.4.3-1.el7 My filesystem disk has 30 GB free space. Cannot start a VM due to an I/O error storage. When tryng to move the disk to another storage domain get this error: Error while executing action: Cannot move Virtual Disk. Low disk space on Storage Domain DATA4. The sum of pre-allocated disk is the total of the storage domain disk. Any idea what can I do to move a disk to other storage domain? Many thanks -- Jose Ferradeira http://www.logicworks.pt

5 5

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

This seems quite odd. What happens when you set custom compatibility mode to 4.2 ? I think it was in Edit -> system -> Advanced options -> Custom compatibility mode -> 4.2 Best Regards, Strahil NikolovOn Jan 12, 2020 13:24, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Hi Strahil, > > > > Unfortunately virtio drives cannot be used. Guest OS is CentOS 5.2, kernel 2.6.18-92.1.17.el5.centos.plusPAE > > > > The network connectivity of the VM is very short (please see below). Till I connect the VM (via spice) network connectivity is already gone. > > tcpdump –I eth0 –v > > tcpdump: listening on eth0, link-type EN10MB ( Ethernet ), capture 96 bytes > > > > 0 packets captured > > 0 packets received by filer > > 0 packets dropped by kernel > > > > From 172.17.17.7 icmp_seq=60 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=61 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=62 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=63 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=64 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=65 Destination Host Unreachable > > 64 bytes from 172.17.16.16: icmp_seq=66 ttl=64 time=2.19 ms > > 64 bytes from 172.17.16.16: icmp_seq=67 ttl=64 time=0.567 ms > > 64 bytes from 172.17.16.16: icmp_seq=68 ttl=64 time=0.339 ms > > 64 bytes from 172.17.16.16: icmp_seq=69 ttl=64 time=0.337 ms > > 64 bytes from 172.17.16.16: icmp_seq=70 ttl=64 time=0.288 ms > > 64 bytes from 172.17.16.16: icmp_seq=71 ttl=64 time=0.305 ms > > 64 bytes from 172.17.16.16: icmp_seq=72 ttl=64 time=0.293 ms > > 64 bytes from 172.17.16.16: icmp_seq=73 ttl=64 time=0.346 ms > > 64 bytes from 172.17.16.16: icmp_seq=74 ttl=64 time=0.359 ms > > 64 bytes from 172.17.16.16: icmp_seq=75 ttl=64 time=0.479 ms > > 64 bytes from 172.17.16.16: icmp_seq=76 ttl=64 time=0.391 ms > > 64 bytes from 172.17.16.16: icmp_seq=77 ttl=64 time=0.302 ms > > 64 bytes from 172.17.16.16: icmp_seq=78 ttl=64 time=0.327 ms > > 64 bytes from 172.17.16.16: icmp_seq=79 ttl=64 time=0.321 ms > > From 172.17.17.7 icmp_seq=125 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=126 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=127 Destination Host Unreachable > > From 172.17.17.7 icmp_seq=128 Destination Host Unreachable > > > > Thank you! > > Best, > > Latcho > > > > > > From: Strahil Nikolov <hunter86_bg(a)yahoo.com> > Sent: Sunday, January 12, 2020 12:52 PM > To: users <users(a)ovirt.org>; Latchezar Filtchev <Latcho(a)aubg.bg> > Subject: Re: [ovirt-users] Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7 > > > > Hi Lacho, > > > > Can you run the virtio drivers ? > > > > They are the most tested in KVM . > > > > Also, can you run a tcpdump (with e1000) and check what is going on when the network disapperead? > > > > Best Regards, > > Strahil Nikolov > > > > В неделя, 12 януари 2020 г., 09:22:52 ч. Гринуич+2, Latchezar Filtchev <latcho(a)aubg.bg> написа: > > > > > > Dear Strahi, > > > > I tried rtl8139. The behavior is the same. > > > > Best, > > Latcho > > > > > > From: Strahil <hunter86_bg(a)yahoo.com> > Sent: Sunday, January 12, 2020 1:19 AM > To: Latchezar Filtchev <Latcho(a)aubg.bg>; users <users(a)ovirt.org> > Subject: Re: [ovirt-users] Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7 > > > > Hi Latcho, > > Most probably it's a bug. > Have you tried virtio and/or rtl-based NIC ? > > As far as I know, CentOS 5 supports Virtio after Kernel >= 2.6.25 . > > Best Regards, > Strahil Nikolov > > On Jan 11, 2020 19:54, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: >> >> Hello Everyone, >> >> >> >> I am aware my guest OS is out of support but I experienced the following: >> >> >> >> oVirt – 4.2.8 (self-hosted engine) >> >> VM – CentOS 5.2; two VNIC’s ( driver used e1000) connected to different VLAN’s; - no issues with network connectivity >> >> >> >> After upgrade to oVirt 4.3.7 >> >> >> >> The same VM starts normally. Network is available for several seconds (10 – 20 pings) and then it disappears. The machine works but no ping to/from VM. When I am returning the same machine (via export domain) to oVirt 4.2.8 environment – it works as expected. >> >> >> >> Can someone advise on this? Any help will be greatly appreciated. >> >> >> >> Thank you. >> >> Best, >> >> Latcho >> >> >> >> >> >> > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DAPMFAA3XZOUJ…

2 1

Re: Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Strahil 12 Jan '20

12 Jan '20

Hi Latcho, Most probably it's a bug. Have you tried virtio and/or rtl-based NIC ? As far as I know, CentOS 5 supports Virtio after Kernel >= 2.6.25 . Best Regards, Strahil NikolovOn Jan 11, 2020 19:54, Latchezar Filtchev <Latcho(a)aubg.bg> wrote: > > Hello Everyone, > > > > I am aware my guest OS is out of support but I experienced the following: > > > > oVirt – 4.2.8 (self-hosted engine) > > VM – CentOS 5.2; two VNIC’s ( driver used e1000) connected to different VLAN’s; - no issues with network connectivity > > > > After upgrade to oVirt 4.3.7 > > > > The same VM starts normally. Network is available for several seconds (10 – 20 pings) and then it disappears. The machine works but no ping to/from VM. When I am returning the same machine (via export domain) to oVirt 4.2.8 environment – it works as expected. > > > > Can someone advise on this? Any help will be greatly appreciated. > > > > Thank you. > > Best, > > Latcho > > > > > >

3 3

Upload ISO to data domain - checksum mismatch on the brick
by Strahil Nikolov 12 Jan '20

12 Jan '20

Hello Community, I'm currently migrating all my ISOs from the ISO domain to the DATA domain but it seems that the checksum (on brick level) mismatches.The problem is that I have managed to upload several ISOs (one of them 4.9GB) without issues. Restarting the ovirt-imageio-proxy.service and later the HostedEngine VM doesn't help. The logs from the imageio proxy indicate no issues. I thought that the gluster could be the problem ,but this is not the case - I tested via copying (using ovirt1) between the 2 mountpoints and the sha512sum matches. Trying to upload via ovirt2 also doesn't help. Any ideas are appreciated. PS: I'm running latest oVirt. Best Regards,Strahil Nikolov

1 0

HCI Disaster Recovery
by Christian Reiss 12 Jan '20

12 Jan '20

Hey folks, - theoretical question, no live data in jeopardy - Let's say a 3-way HCI cluster is up and running, with engine running, all is well. The setup was done via gui, including gluster. Now I would kill a host, poweroff & disk wipe. Simulating a full node failure. The remaining nodes should keep running on (3 copy sync, no arbiter), vms keep running or will be restarted. I would reinstall using the ovirt node installer on the "failed" node. This would net me with a completely empty, no-gluster setup. What is the oVirt way to recover from this point onward? Thanks for your continued support! <3 -Christian. -- with kind regards, mit freundlichen Gruessen, Christian Reiss

2 2

Unstable VM network connectivity after upgrade 4.2.8 to 4.3.7
by Latchezar Filtchev 11 Jan '20

11 Jan '20

Hello Everyone, I am aware my guest OS is out of support but I experienced the following: oVirt - 4.2.8 (self-hosted engine) VM - CentOS 5.2; two VNIC's ( driver used e1000) connected to different VLAN's; - no issues with network connectivity After upgrade to oVirt 4.3.7 The same VM starts normally. Network is available for several seconds (10 - 20 pings) and then it disappears. The machine works but no ping to/from VM. When I am returning the same machine (via export domain) to oVirt 4.2.8 environment - it works as expected. Can someone advise on this? Any help will be greatly appreciated. Thank you. Best, Latcho

1 0

Hosted-engine.metadata header is corrupted
by du_hongyu＠yeah.net 11 Jan '20

11 Jan '20

Hi my Hosted-engine.metadata header is corrupted. I should how to fix it? thanks Regards Hongyu Du

1 0

Re: HCI Disaster Recovery
by Strahil 10 Jan '20

10 Jan '20

It's actually not so easy. The fastest way to recover is just restore from backup. Otherwise the flow should be: 1. Install the new node (new hostname will be easier). 2. Use gluster's replace-brick to change the dead brick with new one. 3. Once ovirt 's integration with gluster detects the change, you will be able to forcefully remove the dead node. 4. Add the newly installed node to the relevant cluster (with or without hosted-engine deployment) 5. Test to move a low-priority VM to the new host. 6. Power up a test VM on the new host to test the functionality. 7. You can make the new node SPM and test snapahots and new vm creation. As you can see , in order to remove a missing node - it should not be part of the Gluster Cluster. Best Regards, Strahil NikolovOn Jan 10, 2020 20:10, Christian Reiss wrote: > > Hey, > > is there really no ovirt native way to restore a single host and bring > it back into the cluster? > > -Chris. > > On 07.01.2020 09:54, Christian Reiss wrote: > > Hey folks, > > > > - theoretical question, no live data in jeopardy - > > > > Let's say a 3-way HCI cluster is up and running, with engine running, > > all is well. The setup was done via gui, including gluster. > > > > Now I would kill a host, poweroff & disk wipe. Simulating a full node > > failure. > > > > The remaining nodes should keep running on (3 copy sync, no arbiter), > > vms keep running or will be restarted. I would reinstall using the ovirt > > node installer on the "failed" node. > > > > This would net me with a completely empty, no-gluster setup. What is the > > oVirt way to recover from this point onward? > > > > Thanks for your continued support! <3 > > -Christian. > > > > -- > Christian Reiss - email(a)christian-reiss.de /"\ ASCII Ribbon > christian(a)reiss.nrw \ / Campaign > X against HTML > XMPP chris(a)alpha-labs.net / \ in eMails > WEB christian-reiss.de, reiss.nrw > > GPG Retrieval http://gpg.christian-reiss.de > GPG ID ABCD43C5, 0x44E29126ABCD43C5 > GPG fingerprint = 9549 F537 2596 86BA 733C A4ED 44E2 9126 ABCD 43C5 > > "It's better to reign in hell than to serve in heaven.", > John Milton, Paradise lost. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GQECSH72L4SMO…

1 0

Hyperconverged setup - storage architecture - scaling
by Leo David 10 Jan '20

10 Jan '20

Hello Everyone, Reading through the document: "Red Hat Hyperconverged Infrastructure for Virtualization 1.5 Automating RHHI for Virtualization deployment" Regarding storage scaling, i see the following statements: *2.7. SCALINGRed Hat Hyperconverged Infrastructure for Virtualization is supported for one node, and for clusters of 3, 6, 9, and 12 nodes.The initial deployment is either 1 or 3 nodes.There are two supported methods of horizontally scaling Red Hat Hyperconverged Infrastructure for Virtualization:* *1 Add new hyperconverged nodes to the cluster, in sets of three, up to the maximum of 12 hyperconverged nodes.* *2 Create new Gluster volumes using new disks on existing hyperconverged nodes.You cannot create a volume that spans more than 3 nodes, or expand an existing volume so that it spans across more than 3 nodes at a time* *2.9.1. Prerequisites for geo-replicationBe aware of the following requirements and limitations when configuring geo-replication:One geo-replicated volume onlyRed Hat Hyperconverged Infrastructure for Virtualization (RHHI for Virtualization) supports only one geo-replicated volume. Red Hat recommends backing up the volume that stores the data of your virtual machines, as this is usually contains the most valuable data.* ------ Also in oVirtEngine UI, when I add a brick to an existing volume i get the following warning: *"Expanding gluster volume in a hyper-converged setup is not recommended as it could lead to degraded performance. To expand storage for cluster, it is advised to add additional gluster volumes." * Those things are raising a couple of questions that maybe for some for you guys are easy to answer, but for me it creates a bit of confusion... I am also referring to RedHat product documentation, because I treat oVirt as production-ready as RHHI is. *1*. Is there any reason for not going to distributed-replicated volumes ( ie: spread one volume across 6,9, or 12 nodes ) ? - ie: is recomanded that in a 9 nodes scenario I should have 3 separated volumes, but how should I deal with the folowing question *2.* If only one geo-replicated volume can be configured, how should I deal with 2nd and 3rd volume replication for disaster recovery *3.* If the limit of hosts per datacenter is 250, then (in theory ) the recomended way in reaching this treshold would be to create 20 separated oVirt logical clusters with 12 nodes per each ( and datacenter managed from one ha-engine ) ? *4.* In present, I have the folowing one 9 nodes cluster , all hosts contributing with 2 disks each to a single replica 3 distributed replicated volume. They where added to the volume in the following order: node1 - disk1 node2 - disk1 ...... node9 - disk1 node1 - disk2 node2 - disk2 ...... node9 - disk2 At the moment, the volume is arbitrated, but I intend to go for full distributed replica 3. Is this a bad setup ? Why ? It oviously brakes the redhat recommended rules... Is there anyone so kind to discuss on these things ? Thank you very much ! Leo -- Best regards, Leo David -- Best regards, Leo David

3 5