July 2019 - Users - oVirt List Archives

Change vDisk ID
by Николаев Алексей 22 Jul '19

22 Jul '19

1 0

Re: Storage domain 'Inactive' but still functional
by Strahil 22 Jul '19

22 Jul '19

If you can give directions (some kind of history) , the dev might try to reproduce this type of issue. If it is reproduceable - a fix can be provided. Based on my experience, if something as used as Linux LVM gets broken, the case is way hard to reproduce. Best Regards, Strahil NikolovOn Jul 22, 2019 10:17, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> wrote: > > Hi, > > Thanks for the tips! I didn't know about 'pvmove', thanks. > > In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. > > Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) > > Thanks, > Martijn. > > > > Op 19-7-2019 om 19:06 schreef Strahil Nikolov: >> >> Hi Martin, >> >> First check what went wrong with the VG -as it could be something simple. >> vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. >> >> If you have Linux boxes - you can add disks from another storage and then pvmove the data inside the VM. Of course , you will need to reinstall grub on the new OS disk , or you won't be able to boot afterwards. >> If possible, try with a test VM before proceeding with important ones. >> >> Backing up the VMs is very important , because working on LVM metadata is quite risky. >> Last time I had such an issue , I was working on clustered LVs which got their PVs "Missing". For me , restore from VG backup fixed the issue - but that might not be always the case. >> >> Just get the vgcfgbackup's output and compare with diff or vimdiff and check what is different. >> >> Sadly, I think that this is more a Linux problem , than an oVirt problem. >> >> Best Regards, >> Strahil Nikolov >> >> В четвъртък, 18 юли 2019 г., 18:51:32 ч. Гринуич+3, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl> написа: >> >> >> Hi! >> >> Thanks. Like I wrote, I have metadata backups from /etc/lvm/backup and -/archive, and I also have the current metadata as it exis

1 0

LiveStoreageMigration failed
by Christoph Köhler 22 Jul '19

22 Jul '19

Hello, I try to migrate a disk of a running vm from gluster 3.12.15 to gluster 3.12.15 but it fails. libGfApi set to true by engine-config. ° taking a snapshot first, is working. Then at engine-log: 2019-07-18 09:29:13,932+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Failed in 'VmReplicateDiskStartVDS' method 2019-07-18 09:29:13,936+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovvirt07 command VmReplicateDiskStartVDS failed: Drive replication error 2019-07-18 09:29:13,936+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand' return value 'StatusOnlyReturn [status=Status [code=55, message=Drive replication error]]' 2019-07-18 09:29:13,936+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] HostName = ovvirt07 2019-07-18 09:29:13,937+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'VmReplicateDiskStartVDSCommand(HostName = ovvirt07, VmReplicateDiskParameters:{hostId='3a7bf85c-e92d-4559-908e-5eed2f5608d4', vmId='3b79d0c0-47e9-47c3-8511-980a8cfe147c', storagePoolId='00000001-0001-0001-0001-000000000311', srcStorageDomainId='e54d835a-d8a5-44ae-8e17-fcba1c54e46f', targetStorageDomainId='4dabb6d6-4be5-458c-811d-6d5e87699640', imageGroupId='d2964ff9-10f7-4b92-8327-d68f3cfd5b50', imageId='62656632-8984-4b7e-8be1-fd2547ca0f98'})' execution failed: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Drive replication error, code = 55 2019-07-18 09:29:13,937+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] FINISH, VmReplicateDiskStartVDSCommand, return: , log id: 5b2afb0b 2019-07-18 09:29:13,937+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Failed VmReplicateDiskStart (Disk 'd2964ff9-10f7-4b92-8327-d68f3cfd5b50' , VM '3b79d0c0-47e9-47c3-8511-980a8cfe147c') 2019-07-18 09:29:13,938+02 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'LiveMigrateDisk' id: '8174c74c-8ab0-49fa-abfc-44d8b7c691e0' with children [03672b60-443b-47ba-834c-ac306d7129d0, 562522fc-6691-47fe-93bf-ef2c45e85676] failed when attempting to perform the next operation, marking as 'ACTIVE' 2019-07-18 09:29:13,938+02 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] EngineException: Drive replication error (Failed with error replicaErr and code 55): org.ovirt.engine.core.common.errors.EngineException: EngineException: Drive replication error (Failed with error replicaErr and code 55) at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.replicateDiskStart(LiveMigrateDiskCommand.java:526) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.performNextOperation(LiveMigrateDiskCommand.java:233) [bll.jar:] at org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:77) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_212] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_212] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2019-07-18 09:29:13,938+02 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Command 'LiveMigrateDisk' id: '8174c74c-8ab0-49fa-abfc-44d8b7c691e0' child commands '[03672b60-443b-47ba-834c-ac306d7129d0, 562522fc-6691-47fe-93bf-ef2c45e85676]' executions were completed, status 'FAILED' 2019-07-18 09:29:15,019+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand' with failure. 2019-07-18 09:29:15,019+02 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [c957e011-37e0-43aa-abe7-9bb633c38c5f] Failed during live storage migration of disk 'd2964ff9-10f7-4b92-8327-d68f3cfd5b50' of vm '3b79d0c0-47e9-47c3-8511-980a8cfe147c', attempting to end replication before deleting the target disk // LiveStorageMigration on gluster - should that work at all? Has someone tried it? Greetings! Christoph Köhler

4 5

Re: Storage domain 'Inactive' but still functional
by Martijn Grendelman 22 Jul '19

22 Jul '19

Hi, Thanks for the tips! I didn't know about 'pvmove', thanks. In the mean time, I managed to get it fixed by restoring the VG metadata on the iSCSI server, so on the underlying Zvol directly, rather than via the iSCSI session on the oVirt host. That allowed me to perform the restore without bringing all VMs down, which was important to me, because if I had to shut down VMs, I was sure I wouldn't be able to restart them before the storage domain was back online. Of course this is a more a Linux problem than an oVirt problem, but oVirt did cause it ;-) Thanks, Martijn. Op 19-7-2019 om 19:06 schreef Strahil Nikolov: Hi Martin, First check what went wrong with the VG -as it could be something simple. vgcfgbackup -f VGname will create a file which you can use to compare current metadata with a previous version. If you have Linux boxes - you can add disks from another storage and then pvmove the data inside the VM. Of course , you will need to reinstall grub on the new OS disk , or you won't be able to boot afterwards. If possible, try with a test VM before proceeding with important ones. Backing up the VMs is very important , because working on LVM metadata is quite risky. Last time I had such an issue , I was working on clustered LVs which got their PVs "Missing". For me , restore from VG backup fixed the issue - but that might not be always the case. Just get the vgcfgbackup's output and compare with diff or vimdiff and check what is different. Sadly, I think that this is more a Linux problem , than an oVirt problem. Best Regards, Strahil Nikolov В четвъртък, 18 юли 2019 г., 18:51:32 ч. Гринуич+3, Martijn Grendelman <Martijn.Grendelman(a)isaac.nl><mailto:Martijn.Grendelman@isaac.nl> написа: Hi! Thanks. Like I wrote, I have metadata backups from /etc/lvm/backup and -/archive, and I also have the current metadata as it exists on disk. What I'm most concerned about, is the proposed procedure. I would create a backup of the VG, but I'm not sure what would be the most sensible way to do it. I could make a new iSCSI target and simply 'dd' the whole disk over, but that would take quite some time (it's 2,5 TB) and there are VMs that can't really be down for that long. And I'm not even sure that dd'ing the disk like that is a sensible strategy. Moving disks out of the domain is currently not possible. oVirt says 'Source Storage Domain is not active'. Thanks, Martijn. Op 18-7-2019 om 17:44 schreef Strahil Nikolov: Can you check the /etc/lvm/backup and /etc/lvm/archive on your SPM host (check the other hosts, just in case you find anything useful) ? Usually LVM makes backup of everything. I would recommend you to: 1. Create a backup of the problematic VG 2. Compare the backup file and a file from backup/archive folders for the same VG Check what is different with diff/vimdiff . It might give you a clue. I had some issues (non-related to oVirt) and restoring the VG from older backup did help me .Still ,any operation on block devices should be considered risky and a proper backup is needed. You could try to move a less important VM's disks out of this storage domain to another one. If it succeeds - then you can evacuate all VMs away before you can start "breaking" the storage domain. Best Regards, Strahil Nikolov В четвъртък, 18 юли 2019 г., 16:59:46 ч. Гринуич+3, Martijn Grendelman <martijn.grendelman(a)isaac.nl><mailto:martijn.grendelman@isaac.nl> написа: Hi, It appears that O365 has trouble delivering mails to this list, so two earlier mails of mine are still somewhere in a queue and may yet be delivered. This mail has all of the content of 3 successive mails. I apologize for this format. Op 18-7-2019 om 11:20 schreef Martijn Grendelman: Op 18-7-2019 om 10:16 schreef Martijn Grendelman: Hi, For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I need some help. Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a disk image off of it, to another domain. This failed, and somewhere in the process, the whole storage domain went to status 'Inactive'. From engine.log: 2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'. 2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2019-07-17 16:30:35,410+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) was deactivated by system because it's not visible by any of the hosts. The thing is, the domain is still functional on all my hosts. It carries over 50 disks, and all involved VMs are up and running, and don't seem to have any problems. Also, 'iscsiadm' on all hosts seems to indiciate that everything is fine with this specific target and reading from the device with dd, or getting its size with 'blockdev' all works without issue. When I try to reactivate the domain, these errors are logged: 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',), code = 358 2019-07-18 09:34:53,648+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) by martijn@-authz On the SPM host, there are errors that indicate problems with the LVM volume group: 2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429', spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48) 2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool] sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78 (sp:1125) 2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=[' /dev/mapper/23536316636393463: Checksum error at offset 2748693688832', " Couldn't read volume group metadata from /dev/mapper/23536316636393463.", ' Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/23536316636393463', ' Failed to scan VG from /dev/mapper/23536316636393463', ' Volume group "875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted: "Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83) I need help getting this storage domain back online. Can anyone here help me? If you need any additional information, please let me know! It appears the VG metadata is corrupt: /dev/mapper/23536316636393463: Checksum error at offset 2748693688832 Couldn't read volume group metadata from /dev/mapper/23536316636393463. Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/23536316636393463 Failed to scan VG from /dev/mapper/23536316636393463 Is this fixable? If so, how? So, I have found some information online, that suggests that PV metadata can be fixed by recreating the PV label using the correct PVID and a backup of the LVM metadata, like so: pvcreate -u <pv_uuid> --restorefile <lvm_metadata_backup> /dev/mapper/23536316636393463 Now I have the following two files: * An LVM metadata backup of yesterday 10:35, about 6 hours before the problem occurred. * The actual metadata found on the PV at offset 2748693688832 (obtained with hexedit on the block device). These are largely the same, but there are differences: * seqno = 1854 in the backup and 1865 in the actual metadata. * 3 logical volumes that are not present in the backup, but are in the actual metadata. I suspect that these are related to snapshots that were created for live storage migration, but I am not sure. In any case, I did NOT create any new disk images on this domain, so that can't be it. Now, support I wanted to try the 'pvcreate' route, then: * what would be the chances of success? Is this procedure at all advisable, or is there an alternative? * which restore file (1854 or 1865) should I use for the restore? * can I do this while the VG is in use? I tried running the command without --force, and it said 'Can't open /dev/mapper/23536316636393463 exclusively. Mounted filesystem?'. I didn't dare to try it with '--force'. I could really use some advice on how to proceed. This are about 36 VMs that have one or more disks on this domain. I could bring them down, although doing so for extended amounts of time would be problematic. I want to be careful, obviously, especially since the actual storage doesn't seem to be impacted at this time. The VMs are all still running without issue, and if I'm about to embark on a dangerous journey that could cause data loss, I need a contingency / recovery plan. Hoping someone can help... Best regards, Martijn Grendelman _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKWOTMY6MKDEQ… -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend. -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend.

1 0

NVMe disk cause kernel panic in installer
by bossman90＠gmail.com 21 Jul '19

21 Jul '19

Hey, I have problems with running oVirt installer on my H370M-ITX mobo. I have installed NVMe disk in it (which I want to use to install oVirt there) and installer crashes with kernel panic: https://photos.app.goo.gl/AY9T93bEs8sCQzRF7 Is there any issues with compatibility oVirt and NVMe?

3 6

Re: NVMe disk cause kernel panic in installer
by Strahil 20 Jul '19

20 Jul '19

Then, you can try a newer version of the kernel from epel repo. Also, check in the CentOS forum - maybe someone had a similar issue. Best Regards, Strahil NikolovOn Jul 20, 2019 03:27, bossman90(a)gmail.com wrote: > > Unfortunately, it didn't help. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2YSRWPT536PAE…

1 0

What values should i give in this ovirt api to get the correct response
by midhun raj 20 Jul '19

20 Jul '19

I am following this link. https://ovirt.org/documentation/admin-guide/chap-Backups_and_Migration.html… I got response for GET /api/vms/4ef8b995-ae58-49d6-8980-ef0fa6688d92/snapshots/1b94ca4e-8c03-4166-8d29-2b831e24b90e/disks HTTP/1.1 Accept: application/xml Content-type: application/xml Identify disk id "Array ( [disk] => Array ( [0] => Array ( [actual_size] => 0 [alias] => GlanceDisk-8700235 [format] => cow [image_id] => a42bda58-7556-4ebe-a54d-03dc0a3b1f54 [propagate_errors] => false [provisioned_size] => 8589934592 [read_only] => false [shareable] => false [sparse] => true [status] => ok [storage_type] => image [wipe_after_delete] => false [snapshot] => Array ( [id] => 1b94ca4e-8c03-4166-8d29-2b831e24b90e ) [storage_domains] => Array ( [storage_domain] => Array ( [0] => Array ( [id] => ad6aea6d-ff90-457f-98eb-99380949e23a ) ) ) [name] => GlanceDisk-8700235 [description] => CentOS 7 Generic Cloud Image v1711 for x86_64 (8700235) [id] => dded245b-4a73-48b4-a1b4-ba98c37490ad ) What is the values that i have to give to the POST request in this api POST /api/vms/22222222-2222-2222-2222-222222222222/disks/ HTTP/1.1 Accept: application/xml Content-type: application/xml <disk id="11111111-1111-1111-1111-111111111111"> <snapshot id="11111111-1111-1111-1111-111111111111"/> <active>true</active> </disk> to replace 1)vms ===>> 22222222-2222-2222-2222-222222222222 2)disk id 11111111-1111-1111-1111-111111111111 3)snapshot id 11111111-1111-1111-1111-111111111111

1 0

Re: Storage domain 'Inactive' but still functional
by Martijn Grendelman 19 Jul '19

19 Jul '19

Hi! Thanks. Like I wrote, I have metadata backups from /etc/lvm/backup and -/archive, and I also have the current metadata as it exists on disk. What I'm most concerned about, is the proposed procedure. I would create a backup of the VG, but I'm not sure what would be the most sensible way to do it. I could make a new iSCSI target and simply 'dd' the whole disk over, but that would take quite some time (it's 2,5 TB) and there are VMs that can't really be down for that long. And I'm not even sure that dd'ing the disk like that is a sensible strategy. Moving disks out of the domain is currently not possible. oVirt says 'Source Storage Domain is not active'. Thanks, Martijn. Op 18-7-2019 om 17:44 schreef Strahil Nikolov: Can you check the /etc/lvm/backup and /etc/lvm/archive on your SPM host (check the other hosts, just in case you find anything useful) ? Usually LVM makes backup of everything. I would recommend you to: 1. Create a backup of the problematic VG 2. Compare the backup file and a file from backup/archive folders for the same VG Check what is different with diff/vimdiff . It might give you a clue. I had some issues (non-related to oVirt) and restoring the VG from older backup did help me .Still ,any operation on block devices should be considered risky and a proper backup is needed. You could try to move a less important VM's disks out of this storage domain to another one. If it succeeds - then you can evacuate all VMs away before you can start "breaking" the storage domain. Best Regards, Strahil Nikolov В четвъртък, 18 юли 2019 г., 16:59:46 ч. Гринуич+3, Martijn Grendelman <martijn.grendelman(a)isaac.nl><mailto:martijn.grendelman@isaac.nl> написа: Hi, It appears that O365 has trouble delivering mails to this list, so two earlier mails of mine are still somewhere in a queue and may yet be delivered. This mail has all of the content of 3 successive mails. I apologize for this format. Op 18-7-2019 om 11:20 schreef Martijn Grendelman: Op 18-7-2019 om 10:16 schreef Martijn Grendelman: Hi, For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I need some help. Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a disk image off of it, to another domain. This failed, and somewhere in the process, the whole storage domain went to status 'Inactive'. From engine.log: 2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'. 2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2019-07-17 16:30:35,410+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) was deactivated by system because it's not visible by any of the hosts. The thing is, the domain is still functional on all my hosts. It carries over 50 disks, and all involved VMs are up and running, and don't seem to have any problems. Also, 'iscsiadm' on all hosts seems to indiciate that everything is fine with this specific target and reading from the device with dd, or getting its size with 'blockdev' all works without issue. When I try to reactivate the domain, these errors are logged: 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',), code = 358 2019-07-18 09:34:53,648+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) by martijn@-authz On the SPM host, there are errors that indicate problems with the LVM volume group: 2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429', spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48) 2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool] sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78 (sp:1125) 2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=[' /dev/mapper/23536316636393463: Checksum error at offset 2748693688832', " Couldn't read volume group metadata from /dev/mapper/23536316636393463.", ' Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/23536316636393463', ' Failed to scan VG from /dev/mapper/23536316636393463', ' Volume group "875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted: "Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83) I need help getting this storage domain back online. Can anyone here help me? If you need any additional information, please let me know! It appears the VG metadata is corrupt: /dev/mapper/23536316636393463: Checksum error at offset 2748693688832 Couldn't read volume group metadata from /dev/mapper/23536316636393463. Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/23536316636393463 Failed to scan VG from /dev/mapper/23536316636393463 Is this fixable? If so, how? So, I have found some information online, that suggests that PV metadata can be fixed by recreating the PV label using the correct PVID and a backup of the LVM metadata, like so: pvcreate -u <pv_uuid> --restorefile <lvm_metadata_backup> /dev/mapper/23536316636393463 Now I have the following two files: * An LVM metadata backup of yesterday 10:35, about 6 hours before the problem occurred. * The actual metadata found on the PV at offset 2748693688832 (obtained with hexedit on the block device). These are largely the same, but there are differences: * seqno = 1854 in the backup and 1865 in the actual metadata. * 3 logical volumes that are not present in the backup, but are in the actual metadata. I suspect that these are related to snapshots that were created for live storage migration, but I am not sure. In any case, I did NOT create any new disk images on this domain, so that can't be it. Now, support I wanted to try the 'pvcreate' route, then: * what would be the chances of success? Is this procedure at all advisable, or is there an alternative? * which restore file (1854 or 1865) should I use for the restore? * can I do this while the VG is in use? I tried running the command without --force, and it said 'Can't open /dev/mapper/23536316636393463 exclusively. Mounted filesystem?'. I didn't dare to try it with '--force'. I could really use some advice on how to proceed. This are about 36 VMs that have one or more disks on this domain. I could bring them down, although doing so for extended amounts of time would be problematic. I want to be careful, obviously, especially since the actual storage doesn't seem to be impacted at this time. The VMs are all still running without issue, and if I'm about to embark on a dangerous journey that could cause data loss, I need a contingency / recovery plan. Hoping someone can help... Best regards, Martijn Grendelman _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKWOTMY6MKDEQ… -- Met vriendelijke groet, Kind regards, [Martijn]<mailto:martijn.grendelman@isaac.nl> Martijn Grendelman Infrastructure Architect T: +31 (0)40 264 94 44 [ISAAC]<https://www.isaac.nl> ISAAC Marconilaan 16 5621 AA Eindhoven The Netherlands T: +31 (0)40 290 89 79 www.isaac.nl<https://www.isaac.nl> [ISAAC #1 Again!]<https://www.isaac.nl/nl/over-ons/nieuws/isaac-news/ISAAC-voor-tweede-keer-n…> Dit e-mail bericht is alleen bestemd voor de geadresseerde(n). Indien dit bericht niet voor u is bedoeld wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. Aan dit bericht kunnen geen rechten worden ontleend.

2 1

Doubt regarding ovirt api
by midhun raj 19 Jul '19

19 Jul '19

following the step 3 of api This is the response i got for GET /api/vms/4ef8b995-ae58-49d6-8980-ef0fa6688d92/snapshots/1b94ca4e-8c03-4166-8d29-2b831e24b90e/disks HTTP/1.1 Accept: application/xml Content-type: application/xml Identify disk id "Array ( [disk] => Array ( [0] => Array ( [actual_size] => 0 [alias] => GlanceDisk-8700235 [format] => cow [image_id] => a42bda58-7556-4ebe-a54d-03dc0a3b1f54 [propagate_errors] => false [provisioned_size] => 8589934592 [read_only] => false [shareable] => false [sparse] => true [status] => ok [storage_type] => image [wipe_after_delete] => false [snapshot] => Array ( [id] => 1b94ca4e-8c03-4166-8d29-2b831e24b90e ) [storage_domains] => Array ( [storage_domain] => Array ( [0] => Array ( [id] => ad6aea6d-ff90-457f-98eb-99380949e23a ) ) ) [name] => GlanceDisk-8700235 [description] => CentOS 7 Generic Cloud Image v1711 for x86_64 (8700235) [id] => dded245b-4a73-48b4-a1b4-ba98c37490ad ) ) ) " What is the values that i have to give to the POST request in this api POST /api/vms/22222222-2222-2222-2222-222222222222/disks/ HTTP/1.1 Accept: application/xml Content-type: application/xml <disk id="11111111-1111-1111-1111-111111111111"> <snapshot id="11111111-1111-1111-1111-111111111111"/> <active>true</active> </disk> to replace 1)vms ===>> 22222222-2222-2222-2222-222222222222 2)disk id 11111111-1111-1111-1111-111111111111 3)snapshot id 11111111-1111-1111-1111-111111111111

1 0

Storage domain 'Inactive' but still functional
by Martijn Grendelman 18 Jul '19

18 Jul '19

Hi, For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I need some help. Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a disk image off of it, to another domain. This failed, and somewhere in the process, the whole storage domain went to status 'Inactive'. From engine.log: 2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'. 2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2019-07-17 16:30:35,410+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) was deactivated by system because it's not visible by any of the hosts. The thing is, the domain is still functional on all my hosts. It carries over 50 disks, and all involved VMs are up and running, and don't seem to have any problems. Also, 'iscsiadm' on all hosts seems to indiciate that everything is fine with this specific target and reading from the device with dd, or getting its size with 'blockdev' all works without issue. When I try to reactivate the domain, these errors are logged: 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:53,631+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',), code = 358 2019-07-18 09:34:53,648+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02 (Data Center ISAAC01) by martijn@-authz On the SPM host, there are errors that indicate problems with the LVM volume group: 2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429', spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48) 2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool] sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78 (sp:1125) 2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed (vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=[' /dev/mapper/23536316636393463: Checksum error at offset 2748693688832', " Couldn't read volume group metadata from /dev/mapper/23536316636393463.", ' Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/23536316636393463', ' Failed to scan VG from /dev/mapper/23536316636393463', ' Volume group "875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) from=::ffff:172.17.1.140,56570, flow_id=197dadec, task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) 2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted: "Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181) 2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83) I need help getting this storage domain back online. Can anyone here help me? If you need any additional information, please let me know! Best regards, Martijn Grendelman ISAAC

3 3

assign vm traffic to a physical NIC and Storage traffic to another NIC
by Erick Perez 18 Jul '19

18 Jul '19

Hi, fresh install I have created a network called "storage" and another called "vms" and both have vlan 102 tag. Only the "vms" network have the check on "vm network". then on Compute---Hosts---hvm001---setup_host_network physical nic enp3s0f0 has ovirtmgmt and vms networks defined physical nic enp3s0f1 has storage network defined Question is how do I tell ovirt that I want the STORAGE traffic to use enp3s0f1 ? I am using NFS data domains and I cannot find a place to tell the network/physical nic my NFS traffic should use. thanks,

2 1

LDAP Group issue with rfc2307bis
by Timmi 18 Jul '19

18 Jul '19

Hi oVirt List, I'm currently working on my new oVirt setup and want to integrate it into our LDAP server. Accounts are working fine but I have problems to get the groups working correctly. The LDAP server is base on ClearOS which is using the rfc2307bis setup. Means I don't have MemberOf inside my users. The user DN is as Member inside the group. I manage that oVirt is able to read the groups while overwriting: search.rfc2307-resolve-groups-memberUid.search-request.filter = &(objectClass=posixGroup)(memberUid=${seq:_rfc2307_uid_encoded}) with search.rfc2307-resolve-groups-memberUid.search-request.filter = &(objectClass=posixGroup)(member=${seq:_rfc2307_dn}) This is working absolutely fine for my admin group in "Administrator Portal". I can asign the group to the system permission "SuperUser" and everything is working great. My problem is with the "VM Portal" I have assigned "PowerUser" rights to a quota and it is possible to login but I receive the following error in the engine.log. 2019-07-18 07:38:12,317+02 ERROR [org.ovirt.engine.core.bll.GetPermissionsForObjectQuery] (default task-5) [a6828f8b-8ded-422f-a216-5e5406d7bf20] Query execution failed due to insufficient permissions. 2019-07-18 07:38:12,319+02 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-5) [] Operation Failed: query execution failed due to insufficient permissions. I'm able to see the group permission in the user details. So I guess that something is already working. But I guess the error is preventing me to have the "create VM" button on the "VM Portal". Would be great if someone could help me out. I'm running the latest 4.3.4 version. Best regards Christoph

1 0

Support and suggestions for 10/25 GbE cards
by Gianluca Merlo 17 Jul '19

17 Jul '19

Hello everyone, I am in the hardware selection phase for an oVirt cluster which has a requirement for 10/25 GbE connectivity. Does anyone have any advice on which adapters are better supported and working on oVirt Node? The choice I have is between: - Intel XXV710 - Mellanox ConnectX-4 It is my first foray into this tier of connectivity, so pardon the naivety of the question. I would just like to hear suggesions from the experience any of you may have had in the field, so I can be better prepared and know which kind of problems I may run into, and if choosing a specific card over the other can spare me some. Thank you in advance for your help Gianluca

1 0

hosted engine, Hyperconverged recovery
by William Kwan 17 Jul '19

17 Jul '19

Hi, I need some direction to make sure we won't make more mistake in recovering a 3-node self hosted engine with Gluster. Someone carelessly, very, wiped out the first few MB of the LVM PV on one node. We changed the quorum and brought up the Gluster and oVirt. We ran vgcfgrestore on the 'bad node' and we can run pvscan, vgscan and lvscan to see the LV's. What should I do next to prevent more damage or corrupt the glusterfs on the other two nodes? Will this work? mkfs the LV's bring up gluster run gluster sync <goodNode>? ThanksWill

3 2

VM says paused, but does not appear to be.
by Jacob Green 17 Jul '19

17 Jul '19

I have a Fibre Channel back end for my oVirt cluster, and one of my VMS is saying "paused due to lack of storage space", however I can confirm that the back end storage has plenty of space and the VM itself does not appear to be paused. I can SSH the VM and perform tasks on the VM. So I am trying to figure out why it would say that, and how to clear it. Also have 20 something other VMS on this cluster and no errors like this on those. I am running engine 4.3.3.7-1.el7 Also I do not fully understand this chart, what I can tell you is that the top storage domain is my Fibre Channel storage, and it has plenty of available space, 35%-40% free And lastly this is what a df -h looks like on the given VM. If anyone has some insight on to what might cause this let me know and ill chase it down. Thank you

1 0

Re: Hardware Used
by Strahil 17 Jul '19

17 Jul '19

As far as I know several orgs are using AMD Ryzen CPUs with oVirt. Ryzen CPUs are quite nice from price/performance perspective. Best Regards, Strahil NikolovOn Jul 17, 2019 15:04, Michal Skrivanek <michal.skrivanek(a)redhat.com> wrote: > > > >> On 16 Jul 2019, at 15:10, Adrian Odendaal <adrian(a)quodes.co.za> wrote: >> > No well we saw that only CPU’s are supported. > > So we want to find out what hardware we need in order to progress. > > Servers with the needed CPU architecture. > > Not sure what you’re looking for. A developer kind of answer - you got that already. If you want anything better I suggest you talk to a vendor who sells oVirt for a living and help you with the right architecture fitting your needs. > List of certified Intel CPUs for RHEL (and hence CentOS) is here [1] and here[2]. We also support IBM POWER8 and POWER9, and if you wanna start something really big, IBM zSystem mainframes as a tech preview;-) > > Thanks, > michal > > [1] https://access.redhat.com/support/policy/intel > [2] https://access.redhat.com/support/policy/amd > >> >> Kind Regards >> >> >> <bcard.jpg> >> >> >> >> >> >> Legal Disclaimer:

2 1

Re: Hardware Used
by Emil Natan 17 Jul '19

17 Jul '19

I think the right place for this question is the ovirt users mailing list. Added. On Mon, Jul 15, 2019 at 10:28 AM <adrian(a)quodes.co.za> wrote: > Hi All > > We are looking into setting up a highly scalable and HA Ovirt > infrastructure but I see only some CPU's are supported. Is this only for th > Hypervisors or for the entire cluster? > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/X5MMJXN67LMBC… > -- Emil Natan RHV/CNV DevOps

3 6

Re: UI exception when changing Cluster in New VM window
by Strahil 17 Jul '19

17 Jul '19

Dear Sharon, I think you mean Marko Vrgotic. Best Regards, Strahil NikolovOn Jul 10, 2019 14:19, Sharon Gratch <sgratch(a)redhat.com> wrote: > > Hi Strahil, > > Based on the stuck trace in the UI log that you sent, I am pretty sure that this UI exception was already fixed on https://gerrit.ovirt.org/#/c/99463/ for oVirt 4.3.4. > So please follow Michal's suggestion to install latest 4.3.5 version for overcoming this issue. > > Thanks, > Sharon > > On Tue, Jul 9, 2019 at 8:19 PM Michal Skrivanek <michal.skrivanek(a)redhat.com> wrote: >> >> can you please try that on 4.3.5 and if it persist open a bug with that ui.log attached? >> >> thanks, >> michal >> >>> On 5 Jul 2019, at 16:06, Vrgotic, Marko <M.Vrgotic(a)activevideo.com> wrote: >>> >>> Hi Strahil, >>> >>> I tried what you said and I am still getting an exception. >>> >>> Correct me if I am wrong, but my expectation is following, when creating new VM: >>> Select cluster >>> Whether I want to use template or create brand new VM from scratch, list of templates and everything else specific to cluster I selected should be loaded >>> >>> Instead, I get the exception and have to refresh/reload the page. >>> >>> In short, when creating new VM, I am currently locked to a default loaded cluster, which in my case is avshared1. >>> >>> To give you bit more insight into configuration, here are the cluster details: >>> >>> <image001.png> >>> >>> Per cluster:

2 1

Re: ovirt_disk and ubuntu issues
by Vrgotic, Marko 16 Jul '19

16 Jul '19

Dear oVIrt, Even though I would like to get some insight into what could be reason this is no working, I did find a workaround: Instead of trying to get Ubuntu disk specified with ovirt_disk size, I used qemu-img resize to increase the disk size before importing it to oVIrt. This works, but it still going to present the problem if User eventually wants to increase for example disk from 40GBto 80GB. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic Sr. System Engineer @ System Administration m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> Date: Wednesday, 10 July 2019 at 16:19 To: "users(a)ovirt.org" <users(a)ovirt.org> Subject: ovirt_disk and ubuntu issues Dear oVirt, I am downloading the ubuntu cloud image 16.04 and or 18.04: - name: "Download base cloud image from server" get_url: url: "{{ image_url }}" checksum: "sha256:{{ image_checksum }}" validate_certs: yes dest: "/tmp/{{ inventory_hostname_short }}.qcow2" delegate_to: localhost creating a 40GB HDD and attaching image to it: - name: "Create oVirt disk with base image (with 40Gb allocated)" ovirt_disk: name: "{{ inventory_hostname_short }}" interface: virtio size: 40GiB format: cow upload_image_path: "/tmp/{{ inventory_hostname_short }}.qcow2" storage_domain: ovirt_production wait: true delegate_to: localhost creating VM afterwards: - name: "Create new Ubuntu VMs from cloud image" delegate_to: localhost ovirt_vm: auth: "{{ ovirt_auth }}" name: "{{ inventory_hostname_short }}" disks: - name: "{{ inventory_hostname_short }}" graphical_console: protocol: vnc serial_console: true usb_support: true soundcard_enabled: false operating_system: "{{ operating_system_type }}" type: server nics: - name: nic1 profile_name: tenant1 interface: virtio nic_on_boot: true cloud_init: host_name: "{{ inventory_hostname }}" user_name: ubuntu authorized_ssh_keys: "{{ ssh_agent_pubkeys.stdout }}" state: "running" cluster: "{{ ovirt_cluster }}" when: inventory_hostname in groups['ubuntu-baker'] When VM gets created, I can see in oVIrt VM details disk created is 40GB. Executing df -h, gives me following: root@av3-ubuntu-18-base:/home/ubuntu# df -h Filesystem Size Used Avail Use% Mounted on udev 447M 0 447M 0% /dev tmpfs 92M 696K 92M 1% /run /dev/vda1 2.0G 1.3G 706M 65% / tmpfs 460M 0 460M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 460M 0 460M 0% /sys/fs/cgroup /dev/vda15 105M 3.6M 101M 4% /boot/efi tmpfs 92M 0 92M 0% /run/user/1000 Initially I thought growpart or resize2fs is not triggered, but then running dmesg or fdisk /dev/vda, told me that physical disk size is still only size of the downloaded ubuntu cloud image. Disk /dev/vda: 2.2 GiB, 2361393152 bytes, 4612096 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Is this related to ovirt_disk module, or ubuntu and ovirt_disk, since I do not have this behavior with CentOS 7 images? Can you advise how to proceed, in case I am missing some configuration parameter or command to be run? The following Ubuntu images are used: ubuntu-16: image_url=https://cloud-images.ubuntu.com/xenial/current/xenial-server-clou… image_checksum=fda868058586b129c7fdb6472fe575e911f7c67551a6dc75966f2ec02201bdae ubuntu-18: image_url=https://cloud-images.ubuntu.com/bionic/current/bionic-server-clou… image_checksum=7d2b90022a169119d7726c0fefa1713acbead7cc36d282c879896fd89c5a6663 Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic Sr. System Engineer @ System Administration m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> tel. +31 (0)35 677 4131

2 4

Moving Hosted Engine Storage
by Dan Poltawski 16 Jul '19

16 Jul '19

Hello, I've read various posts[1] on this list, but I must confess that I am still not entirely clear on the process for moving a hosted engine to a new storage domain (in my case I want to move from an existing NFS server to a new iSCSI target). Some of what i've read make me slightly concerned it's a risky operation and in my situation I am just prototyping and retaining the existing engine will be saving some time rather than mission critical. Is anyone able to outline the steps to me? thanks, Dan [1] https://lists.ovirt.org/pipermail/users/2017-June/082466.html ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

2 1

Re: Update 4.2.8 --> 4.3.5
by Strahil 16 Jul '19

16 Jul '19

I'm addding gluster-users as I'm not sure if you can go gluster v3 -> v6 directly. Theoretically speaking , there should be no problem - but I don't know if you will observe any issues. @Gluster-users, Can someone share their thoughts about v3 to v6 migration ? Best Regards, Strahil NikolovOn Jul 11, 2019 14:05, Christoph Köhler wrote: > > Hello! > > We have a 4.2.8 environment with some managed gluster-volumes as storage > domains and we want to go up to 4.3.5. > > How is that procedure especially with the gluster nodes in ovirt that > are running 3.12.15? My fear is on the jump to gluster 6. Do the cluster > work if the first node (of three) is upgraded? And what about the > sequence - first the hypervisors or first gluster nodes? > > Is there anyone who had done this? > > Greetings! > Christoph Köhler > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VWFS6YUKA77VP…

3 2

Can't oepn console to VM
by csi-laser＠cisco.com 16 Jul '19

16 Jul '19

Hi, When trying to open a console for my VM , I'm getting this error : [cid:image001.png@01D53732.4E21AD20] Can you please help ? Thanks in advance

1 1

VM status codes
by Mitja Mihelič 15 Jul '19

15 Jul '19

Hi! We are graphing certain parameters we collect from the engine database. We use following query to get the count of running VMs: select count(*),cluster.name as cluster from vms, cluster where vms.cluster_id = cluster.cluster_id and vms.status = 1 group by cluster; It would help extremely if we knew what the numeric VM status codes mean. For example, I know that "status = 1" means the VM is running. The ones we can cause through the web interface are easy enough to figure out. But there are error states, that cannot be produced by standard use. For instance, when the event log says "VM foobar has been paused due to storage I/O problem." the following states get set in the DB (Query used: select * from vm_dynamic where vm_guid =(select vm_guid from vm_static where vm_name = 'foobar';): status | 4 exit_status | 0 pause_status | 2 guest_agent_status | 0 What do they mean? If you could point me to a table somewhere with the state descriptions it would be extremely helpful. Kind regards, Mitja Mihelič

2 2

SSH Authorized_keys
by Vrgotic, Marko 15 Jul '19

15 Jul '19

Dear oVIrt, Would it be possible to utilize SSH Pub key from User config [cid:image001.png@01D53B2B.0E387070] In such a way that it is offered or autoloaded into Cloud-init SSH Authorized keys field when exists? This would bring one less action required before VM can be created. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

1 0

Re: ovirt-engine-appliance ova
by Strahil 15 Jul '19

15 Jul '19

Exactly. Ovirt (and RHV respectively) can be installed as: 1. VM in an oVirt Cluster 2. Separate machine(s) In point 2 , the Virt admin should take care about the high availability of the engine. So, you can install CentOS7 and then deploy the engine's software ontop of it. Extracting the OVA will not provide you with fully working engine, as the deployment scripts on the hosts will not inject the necessary data on remote system. (Only on local) Best Regards, Strahil NikolovOn Jul 15, 2019 04:08, Jingjie Jiang <jingjie.jiang(a)oracle.com> wrote: > > > On 7/13/2019 6:28 AM, Strahil Nikolov wrote: > > I still don't get why you need the OVA . > Are you trying to extract and put it into another virtualization. If this is your intention - better install it as if it was a standalone engine. > > Yes, I am trying to build ova with different OS. > > What do you mean install it as standalone engine? > > install VM first then install as ovirt engine? > > > Thanks, > > Jingjie > > >> >> Best Regards, >> Strahil Nikolov. >> >> В петък, 12 юли 2019 г., 21:48:35 ч. Гринуич+3, Jingjie Jiang <jingjie.jiang(a)oracle.com> написа: >> >> >> Hi Strahil, >> >> Thanks for your reply. >> >> Can you share the procedure of creating ovirt-engine-appliance ova? >> >> >> Thanks, >> >> Jingjie >> >> >> On 7/12/19 12:52 AM, Strahil wrote: >> >> Based on my experience - the OVA contains the xml and the actual disk of the hosted engine. >> Then the deployment starts locally the VM and populates the necessary data in it >> Once it's over , the deployment shuts down and copies the disk of that local VM, undefines it and then ovirt's ha agents are being configured - so they can mount the shared storage and power up the VM (special tar /OVMF/ file on shared storage has the agent configuration file). >> >> So , in the ova there should be a template VM + the xml config (cpus, ram, devices, etc) . >> I would be surprised if there is something else in it. >> >> Best Regards, >> Strahil Nikolov >> >> On Jul 11, 2019 23:39, Jingjie Jiang <jingjie.jiang(a)oracle.com> wrote:

1 0