Hosted Engine VM not imported

Hi, I have just migrated my dev cluster to the latest master, reinstalling the engine VM and reimporting from a previous backup. I'm trying with 4.3.0-0.0.master.20180222192611.git01e6ace.el7.centos I had a few problems: - the documentation seems to be outdated, and I just find by searching the archives that it's needed to add the two (undocumented) options --he-remove-storage-vm --he-remove-hosts - despite the fact I selected "No" to running the engine-setup command in the VM (the ovirt appliance), the engine-setup is executed when running hosted-engine --deploy, and as a result the procedure does not stop allowing to reload the db backup. The only way I found was to put the hosted-engine in global maintenance mode, stop the ovirt-engine, do an engine-cleanup and reload the db, then it's possible to add the first host in the GUI, but must be done manually - after it's all done, I can see the hosted_storage is imported, but the HostedEngine is not imported, and in the Events I see messages like this: VDSM atlas-svc-18 command GetVolumeInfoVDS failed: Image path does not exist or cannot be accessed/created: (u'/rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/c815ec3f-6e31-4b08-81be-e515e803edce',) the path here is clearly wrong, it should be /rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/b7bc6468-438c-47e7-b7a4-7ed06b786da0/c815ec3f-6e31-4b08-81be-e515e803edce, and I see the hosted_engine.conf in the shared storage has it correctly set as vm_disk_id=b7bc6468-438c-47e7-b7a4-7ed06b786da0. Any hint on what is not allowing the HostedEngine to be imported? I didn't find a way to add other hosted engine nodes if the HE VM is not imported in the cluster, like we were used in the past with the CLI using hosted-engine --deploy on multiple hosts. Thanks for any help, Alessandro

Hi, after checking the engine.log I see a bunch of error like this too: 2018-02-26 03:22:06,806+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] HostName = atlas-svc-18 2018-02-26 03:22:06,806+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] Command 'GetVolumeInfoVDSCommand(HostName = atlas-svc-18, GetVolumeInfoVDSCommandParameters:{hostId='b18c40d8-7932-4b5d-995e-8ebc5ab2e3e2', storagePoolId='00000001-0001-0001-0001-000000000056', storageDomainId='f02d7d5d-1459-48b8-bf27-4225cdfdce23', imageGroupId='c815ec3f-6e31-4b08-81be-e515e803edce', imageId='c815ec3f-6e31-4b08-81be-e515e803edce'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetVolumeInfoVDS, error = Image path does not exist or cannot be accessed/created: (u'/rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/c815ec3f-6e31-4b08-81be-e515e803edce',), code = 254 2018-02-26 03:22:06,806+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] FINISH, GetVolumeInfoVDSCommand, log id: 38adaef0 2018-02-26 03:22:06,806+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] Failed to get the volume information, marking as FAILED 2018-02-26 03:22:06,806+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] FINISH, GetImageInfoVDSCommand, log id: 3ad29b91 2018-02-26 03:22:06,806+01 WARN [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] Validation of action 'ImportVm' failed for user SYSTEM. Reasons: VAR__ACTION__IMPORT,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IMAGE_DOES_NOT_EXIST 2018-02-26 03:22:06,807+01 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] Lock freed to object 'EngineLock:{exclusiveLocks='[HostedEngine=VM_NAME, 235b91ce-b6d8-44c6-ac26-791ac3946727=VM]', sharedLocks='[235b91ce-b6d8-44c6-ac26-791ac3946727=REMOTE_VM]'}' 2018-02-26 03:22:06,807+01 ERROR [org.ovirt.engine.core.bll.HostedEngineImporter] (EE-ManagedThreadFactory-engine-Thread-97153) [796a8bc5] Failed importing the Hosted Engine VM Any help? Thanks, Alessandro Il 24/02/18 14:32, Alessandro De Salvo ha scritto:
Hi,
I have just migrated my dev cluster to the latest master, reinstalling the engine VM and reimporting from a previous backup. I'm trying with 4.3.0-0.0.master.20180222192611.git01e6ace.el7.centos
I had a few problems:
- the documentation seems to be outdated, and I just find by searching the archives that it's needed to add the two (undocumented) options --he-remove-storage-vm --he-remove-hosts
- despite the fact I selected "No" to running the engine-setup command in the VM (the ovirt appliance), the engine-setup is executed when running hosted-engine --deploy, and as a result the procedure does not stop allowing to reload the db backup. The only way I found was to put the hosted-engine in global maintenance mode, stop the ovirt-engine, do an engine-cleanup and reload the db, then it's possible to add the first host in the GUI, but must be done manually
- after it's all done, I can see the hosted_storage is imported, but the HostedEngine is not imported, and in the Events I see messages like this:
VDSM atlas-svc-18 command GetVolumeInfoVDS failed: Image path does not exist or cannot be accessed/created: (u'/rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/c815ec3f-6e31-4b08-81be-e515e803edce',)
the path here is clearly wrong, it should be /rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/b7bc6468-438c-47e7-b7a4-7ed06b786da0/c815ec3f-6e31-4b08-81be-e515e803edce, and I see the hosted_engine.conf in the shared storage has it correctly set as vm_disk_id=b7bc6468-438c-47e7-b7a4-7ed06b786da0.
Any hint on what is not allowing the HostedEngine to be imported? I didn't find a way to add other hosted engine nodes if the HE VM is not imported in the cluster, like we were used in the past with the CLI using hosted-engine --deploy on multiple hosts.
Thanks for any help,
Alessandro
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Sat, Feb 24, 2018 at 2:32 PM, Alessandro De Salvo < Alessandro.DeSalvo@roma1.infn.it> wrote:
Hi,
I have just migrated my dev cluster to the latest master, reinstalling the engine VM and reimporting from a previous backup. I'm trying with 4.3.0-0.0.master.20180222192611.git01e6ace.el7.centos
I had a few problems:
- the documentation seems to be outdated, and I just find by searching the archives that it's needed to add the two (undocumented) options --he-remove-storage-vm --he-remove-hosts
- despite the fact I selected "No" to running the engine-setup command in the VM (the ovirt appliance), the engine-setup is executed when running hosted-engine --deploy, and as a result the procedure does not stop allowing to reload the db backup. The only way I found was to put the hosted-engine in global maintenance mode, stop the ovirt-engine, do an engine-cleanup and reload the db, then it's possible to add the first host in the GUI, but must be done manually
- after it's all done, I can see the hosted_storage is imported, but the HostedEngine is not imported, and in the Events I see messages like this:
VDSM atlas-svc-18 command GetVolumeInfoVDS failed: Image path does not exist or cannot be accessed/created: (u'/rhev/data-center/mnt/glust erSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/ f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/c815ec3f-6e31- 4b08-81be-e515e803edce',)
the path here is clearly wrong, it should be /rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn. it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/ images/b7bc6468-438c-47e7-b7a4-7ed06b786da0/c815ec3f-6e31-4b08-81be-e515e803edce, and I see the hosted_engine.conf in the shared storage has it correctly set as vm_disk_id=b7bc6468-438c-47e7-b7a4-7ed06b786da0.
Any hint on what is not allowing the HostedEngine to be imported? I didn't find a way to add other hosted engine nodes if the HE VM is not imported in the cluster, like we were used in the past with the CLI using hosted-engine --deploy on multiple hosts.
Ciao Alessandro, with 4.2.1 we introduced a new deployment flow for hosted-engine based on ansible. In this new flow we run a local VM with a running engine and we use that engine to create a storage domain and a VM there. At the end we shutdown the locally running engine and we move it's disk over the disk of the VM created by the engine on the shared storage. At this point we don't need anymore the autoimport process since the engine migrated there already contains the engine VM and its storage domain. We have an RFE, for this new flow, to add a mechanism to inject an existing engine backup to be automatically restored before executing engine-setup for migration/disaster-recovery scenarios. Unfortunately it's still not ready but we have an hook mechanism to have hosted-engine-setup executing custom ansible tasks before running engine setup; we have an example in /usr/share/ovirt-hosted-engine-setup/ansible/hooks/enginevm_before_engine_setup/enginevm_before_engine_setup.yml.example Otherwise the old flow is still there, you have just to add --noansible and everything should work as in the past.
Thanks for any help,
Alessandro
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

This is a multi-part message in MIME format. --------------67ED3AA47939DF7515991C15 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Ciao Simone, many thanks. So, how are we supposed to use those hooks? Should we just create a file /usr/share/ovirt-hosted-engine-setup/ansible/hooks/enginevm_before_engine_setup/enginevm_before_engine_setup.yml with the instructions to restore? Do you have an example for doing that? For the moment I think I'll stick to the old procedure by calling --noansible, as you suggest. I think the documenttation should be updated anyways, at least to add the --he-remove-storage-vm and --he-remove-hosts options, as well as the new procedure and the override with --noansible. Also, wouldn't it be safer to stick to the old procedure until the new one is fully operational? Or maybe at least a warning to the user, otherwise no one will ever be able to restorage a db and have it all functional with the default options. Thanks, Alessandro Il 26/02/18 18:17, Simone Tiraboschi ha scritto:
On Sat, Feb 24, 2018 at 2:32 PM, Alessandro De Salvo <Alessandro.DeSalvo@roma1.infn.it <mailto:Alessandro.DeSalvo@roma1.infn.it>> wrote:
Hi,
I have just migrated my dev cluster to the latest master, reinstalling the engine VM and reimporting from a previous backup. I'm trying with 4.3.0-0.0.master.20180222192611.git01e6ace.el7.centos
I had a few problems:
- the documentation seems to be outdated, and I just find by searching the archives that it's needed to add the two (undocumented) options --he-remove-storage-vm --he-remove-hosts
- despite the fact I selected "No" to running the engine-setup command in the VM (the ovirt appliance), the engine-setup is executed when running hosted-engine --deploy, and as a result the procedure does not stop allowing to reload the db backup. The only way I found was to put the hosted-engine in global maintenance mode, stop the ovirt-engine, do an engine-cleanup and reload the db, then it's possible to add the first host in the GUI, but must be done manually
- after it's all done, I can see the hosted_storage is imported, but the HostedEngine is not imported, and in the Events I see messages like this:
VDSM atlas-svc-18 command GetVolumeInfoVDS failed: Image path does not exist or cannot be accessed/created: (u'/rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/c815ec3f-6e31-4b08-81be-e515e803edce',)
the path here is clearly wrong, it should be /rhev/data-center/mnt/glusterSD/atlas-fsserv-07.roma1.infn.it:_atlas-engine-02/f02d7d5d-1459-48b8-bf27-4225cdfdce23/images/b7bc6468-438c-47e7-b7a4-7ed06b786da0/c815ec3f-6e31-4b08-81be-e515e803edce, and I see the hosted_engine.conf in the shared storage has it correctly set as vm_disk_id=b7bc6468-438c-47e7-b7a4-7ed06b786da0.
Any hint on what is not allowing the HostedEngine to be imported? I didn't find a way to add other hosted engine nodes if the HE VM is not imported in the cluster, like we were used in the past with the CLI using hosted-engine --deploy on multiple hosts.
Ciao Alessandro, with 4.2.1 we introduced a new deployment flow for hosted-engine based on ansible. In this new flow we run a local VM with a running engine and we use that engine to create a storage domain and a VM there. At the end we shutdown the locally running engine and we move it's disk over the disk of the VM created by the engine on the shared storage. At this point we don't need anymore the autoimport process since the engine migrated there already contains the engine VM and its storage domain.
We have an RFE, for this new flow, to add a mechanism to inject an existing engine backup to be automatically restored before executing engine-setup for migration/disaster-recovery scenarios. Unfortunately it's still not ready but we have an hook mechanism to have hosted-engine-setup executing custom ansible tasks before running engine setup; we have an example in /usr/share/ovirt-hosted-engine-setup/ansible/hooks/enginevm_before_engine_setup/enginevm_before_engine_setup.yml.example
Otherwise the old flow is still there, you have just to add --noansible and everything should work as in the past.
Thanks for any help,
Alessandro
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
--------------67ED3AA47939DF7515991C15 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Ciao Simone,</p> <p>many thanks. So, how are we supposed to use those hooks? Should we just create a file /usr/share/ovirt-hosted-engine-setup/ansible/hooks/enginevm_before_engine_setup/enginevm_before_engine_setup.yml with the instructions to restore? Do you have an example for doing that?</p> <p>For the moment I think I'll stick to the old procedure by calling --noansible, as you suggest.</p> <p>I think the documenttation should be updated anyways, at least to add the --he-remove-storage-vm and --he-remove-hosts options, as well as the new procedure and the override with --noansible. Also, wouldn't it be safer to stick to the old procedure until the new one is fully operational? Or maybe at least a warning to the user, otherwise no one will ever be able to restorage a db and have it all functional with the default options.</p> <p>Thanks,</p> <p><br> </p> <p> Alessandro<br> </p> <br> <div class="moz-cite-prefix">Il 26/02/18 18:17, Simone Tiraboschi ha scritto:<br> </div> <blockquote type="cite" cite="mid:CAN8-ONqTkkui-Gih4gCpT3rv+rCaBPqrvwzD83RbGe57Tp4_Qg@mail.gmail.com"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Sat, Feb 24, 2018 at 2:32 PM, Alessandro De Salvo <span dir="ltr"><<a href="mailto:Alessandro.DeSalvo@roma1.infn.it" target="_blank" moz-do-not-send="true">Alessandro.DeSalvo@roma1.infn.it</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br> <br> I have just migrated my dev cluster to the latest master, reinstalling the engine VM and reimporting from a previous backup. I'm trying with 4.3.0-0.0.master.2018022219261<wbr>1.git01e6ace.el7.centos<br> <br> I had a few problems:<br> <br> - the documentation seems to be outdated, and I just find by searching the archives that it's needed to add the two (undocumented) options --he-remove-storage-vm --he-remove-hosts<br> <br> - despite the fact I selected "No" to running the engine-setup command in the VM (the ovirt appliance), the engine-setup is executed when running hosted-engine --deploy, and as a result the procedure does not stop allowing to reload the db backup. The only way I found was to put the hosted-engine in global maintenance mode, stop the ovirt-engine, do an engine-cleanup and reload the db, then it's possible to add the first host in the GUI, but must be done manually<br> <br> - after it's all done, I can see the hosted_storage is imported, but the HostedEngine is not imported, and in the Events I see messages like this:<br> <br> VDSM atlas-svc-18 command GetVolumeInfoVDS failed: Image path does not exist or cannot be accessed/created: (u'/rhev/data-center/mnt/glust<wbr>erSD/atlas-fsserv-07.roma1.<wbr>infn.it:_atlas-engine-02/<wbr>f02d7d5d-1459-48b8-bf27-4225cd<wbr>fdce23/images/c815ec3f-6e31-<wbr>4b08-81be-e515e803edce',)<br> <br> the path here is clearly wrong, it should be /rhev/data-center/mnt/glusterS<wbr>D/atlas-fsserv-07.roma1.infn.<wbr>it:_atlas-engine-02/f02d7d5d-<wbr>1459-48b8-bf27-4225cdfdce23/<wbr>images/b7bc6468-438c-47e7-<wbr>b7a4-7ed06b786da0/c815ec3f-<wbr>6e31-4b08-81be-e515e803edce, and I see the hosted_engine.conf in the shared storage has it correctly set as vm_disk_id=b7bc6468-438c-47e7-<wbr>b7a4-7ed06b786da0.<br> <br> <br> Any hint on what is not allowing the HostedEngine to be imported? I didn't find a way to add other hosted engine nodes if the HE VM is not imported in the cluster, like we were used in the past with the CLI using hosted-engine --deploy on multiple hosts.<br> </blockquote> <div><br> </div> <div>Ciao Alessandro,</div> <div>with 4.2.1 we introduced a new deployment flow for hosted-engine based on ansible.</div> <div>In this new flow we run a local VM with a running engine and we use that engine to create a storage domain and a VM there.</div> <div>At the end we shutdown the locally running engine and we move it's disk over the disk of the VM created by the engine on the shared storage. At this point we don't need anymore the autoimport process since the engine migrated there already contains the engine VM and its storage domain.</div> <div><br> </div> <div>We have an RFE, for this new flow, to add a mechanism to inject an existing engine backup to be automatically restored before executing engine-setup for migration/disaster-recovery scenarios.</div> <div>Unfortunately it's still not ready but we have an hook mechanism to have hosted-engine-setup executing custom ansible tasks before running engine setup; we have an example in /usr/share/ovirt-hosted-engine-setup/ansible/hooks/enginevm_before_engine_setup/enginevm_before_engine_setup.yml.example</div> <div><br> </div> <div>Otherwise the old flow is still there, you have just to add --noansible and everything should work as in the past.</div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> Thanks for any help,<br> <br> <br> Alessandro<br> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a href="mailto:Users@ovirt.org" target="_blank" moz-do-not-send="true">Users@ovirt.org</a><br> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </blockquote> </div> <br> </div> </div> </blockquote> <br> </body> </html> --------------67ED3AA47939DF7515991C15--
participants (2)
-
Alessandro De Salvo
-
Simone Tiraboschi