Hi, We are planning to add couple of verbs to vdsm in order to import external VMs from VMWare esx server directly to data domain. This ability will also be reflected in oVirt engine and will enable user to view and select VMs from esx server, currently if oVirt user want to import a VM he needs to invoke a virt-v2v command line that will transfer the VM to an export domain and then import the same VM from export domain to data domain, Our goal is to eliminate the additional copy and simplify the process. In order to achive that we need to add two verbs for vdsm: 1. getExternalVmList(uri): This call is for listing all VMs that are raised in a specified esx server given its uri. 2. convertExternalVm(): The actual call to virt-v2v for copy and convert the VM to data domain, importVm will be called after the oVirt engine create the right image structure (via createImage) The virt-v2v handling verbs and other needed logic will be presented in a new file vdsm/v2v.py More details on virt-v2v integration can be found in its feature page: http://www.ovirt.org/Features/virt-v2v_Integration

Thanks, Shahar Havivi.

To expand on the previous email, the web page is quite vague about what precise sources you want to import from. The ones supported by virt-v2v are: - VMware live vCenter: I assume you would want this. (Note ESXi is mentioned in the web page, but virt-v2v does not support import from ESXi directly) - OVA file: As Shahar wrote this bit, I'll assume you'll want to use it. - RHEL 5 Xen: SSH key management is going to be a problem here. - Local disk files: Seems unlikely you'll want this. - Citrix Xen, Hyper-V, etc.: Requires development work. - Physical machines VMware vCenter -------------- See: http://libguestfs.org/virt-v2v.1.html#input-from-vmware-vcenter-server The user will need to specify: - hostname of VMware vCenter server - optional port number - username - password - name of VMware Datacenter - hostname of VMware ESXi server [may be able to relax this in future] - whether or not to check HTTPS certs - name of the guest As detailed in the link above, there are various virsh commands you can run from VDSM to establish whether conversion is likely to work, and to get a list of guests which you can present to the user. OVA file -------- The user will need to specify: - OVA file (eg. location) I've no idea how the file will end up being accessible to VDSM. NFS mount? RHEL 5 Xen ---------- See: http://libguestfs.org/virt-v2v.1.html#input-from-rhel-5-xen I think we're going to have a problem with ssh keys, because this mode currently only works when the virt-v2v client can get passwordless access to the RHEL 5 Xen server using ssh-agent (note that Kerberos does not work). We might fix this in future, but at the moment you're somehow going to have to ensure that the oVirt node has a private key which is listed in the authorized_keys in the RHEL 5 Xen server. I simply cannot imagine a user interface that is going to make this usable. I should also say that few people care about RHEL 5 Xen imports. Citrix Xen imports OTOH, certainly, but those were out of scope for current upstream development for RHEL 7.1. Anyway, the user will have to specify: - hostname of RHEL 5 Xen server - username - password - name of the guest There are various virsh commands you can run from VDSM to establish whether conversion is likely to work, and to get a list of guests which you can present to the user. Physical machines ----------------- See also: http://libguestfs.org/virt-p2v.1.html [Update from previous email: I have now updated this page to contain the latest documentation.] Virt-v2v can import physical machines, but you have to boot the separate virt-p2v ISO on the physical machine, and generally speaking the process is directed from the physical machine. This makes it quite unlike the other input sources. (Users will still be able to use the self-boot + '-o rhev' method, ie. import into the Export Storage Domain, as with RHEL 6). If oVirt can (or does?) run a PXE server, then it's possible to serve a virt-p2v bootable image to physical machines, and it's also possible to automate it from the PXE server side. [http://libguestfs.org/virt-p2v.1.html#kernel-command-line-configuration] Environment variables --------------------- You may also want to set the following environment variables before running virt-v2v: LIBGUESTFS_BACKEND=direct This is currently required for VMware and RHEL 5 Xen conversions owing to a bug in libvirt (1134592) that was today pushed to RHEL 7.2. You should be able to drop this once libvirt is fixed. TMPDIR=... You might want to set this to control where large temporary files are created during the v2v process. Other environment variables that may affect virt-v2v are discussed on both of these pages: http://libguestfs.org/guestfs.3.html#environment-variables http://libguestfs.org/virt-v2v.1.html#environment-variables A note on the output mode ------------------------- You're going to be using '-o vdsm -os /storage_domain' and the other options: '--vdsm-image-uuid UUID', '--vdsm-vol-uuid UUID', '--vdsm-vm-uuid UUID' [http://libguestfs.org/virt-v2v.1.html#options] Implicit in the contract with '-o vdsm' is that VDSM is responsible for creating the output directories, running virt-v2v as user 36:36, and deleting the output directories if virt-v2v fails.

In this output mode virt-v2v doesn't create or clean up after itself or try to chown files (unlike '-o rhev'). HTH, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

On 09/30/2014 04:18 PM, Arik Hadas wrote: > > > ----- Original Message ----- >> To expand on the previous email, the web page is quite vague about >> what precise sources you want to import from. >> >> The ones supported by virt-v2v are: >> >> - VMware live vCenter: I assume you would want this. (Note ESXi is >> mentioned in the web page, but virt-v2v does not support import >> from ESXi directly) >> >> - OVA file: As Shahar wrote this bit, I'll assume you'll want to use >> it. >> >> - RHEL 5 Xen: SSH key management is going to be a problem here. >> >> - Local disk files: Seems unlikely you'll want this. >> >> - Citrix Xen, Hyper-V, etc.: Requires development work. >> >> - Physical machines >> >> VMware vCenter >> -------------- >> >> See: >> http://libguestfs.org/virt-v2v.1.html#input-from-vmware-vcenter-server >> >> The user will need to specify: >> - hostname of VMware vCenter server >> - optional port number >> - username >> - password >> - name of VMware Datacenter >> - hostname of VMware ESXi server [may be able to relax this in future] >> - whether or not to check HTTPS certs >> - name of the guest >> >> As detailed in the link above, there are various virsh commands you >> can run from VDSM to establish whether conversion is likely to work, >> and to get a list of guests which you can present to the user. >> >> OVA file >> -------- >> >> The user will need to specify: >> - OVA file (eg. location) >> >> I've no idea how the file will end up being accessible to VDSM. NFS >> mount? >> >> RHEL 5 Xen >> ---------- >> >> See: http://libguestfs.org/virt-v2v.1.html#input-from-rhel-5-xen >> >> I think we're going to have a problem with ssh keys, because this mode >> currently only works when the virt-v2v client can get passwordless >> access to the RHEL 5 Xen server using ssh-agent (note that Kerberos >> does not work). We might fix this in future, but at the moment you're >> somehow going to have to ensure that the oVirt node has a private key >> which is listed in the authorized_keys in the RHEL 5 Xen server. I >> simply cannot imagine a user interface that is going to make this >> usable. >> >> I should also say that few people care about RHEL 5 Xen imports. >> Citrix Xen imports OTOH, certainly, but those were out of scope for >> current upstream development for RHEL 7.1. >> >> Anyway, the user will have to specify: >> >> - hostname of RHEL 5 Xen server >> - username >> - password >> - name of the guest >> >> There are various virsh commands you can run from VDSM to establish >> whether conversion is likely to work, and to get a list of guests >> which you can present to the user. >> >> Physical machines >> ----------------- >> >> See also: http://libguestfs.org/virt-p2v.1.html >> [Update from previous email: I have now updated this page to contain >> the latest documentation.] >> >> Virt-v2v can import physical machines, but you have to boot the >> separate virt-p2v ISO on the physical machine, and generally speaking >> the process is directed from the physical machine. >> >> This makes it quite unlike the other input sources. (Users will still >> be able to use the self-boot + '-o rhev' method, ie. import into the >> Export Storage Domain, as with RHEL 6). >> >> If oVirt can (or does?) run a PXE server, then it's possible to serve a >> virt-p2v bootable image to physical machines, and it's also possible >> to automate it from the PXE server side. >> [http://libguestfs.org/virt-p2v.1.html#kernel-command-line-configuration] >> >> Environment variables >> --------------------- >> >> You may also want to set the following environment variables before >> running virt-v2v: >> >> LIBGUESTFS_BACKEND=direct >> >> This is currently required for VMware and RHEL 5 Xen conversions owing >> to a bug in libvirt (1134592) that was today pushed to RHEL 7.2. You >> should be able to drop this once libvirt is fixed. >> >> TMPDIR=... >> >> You might want to set this to control where large temporary files are >> created during the v2v process. >> >> Other environment variables that may affect virt-v2v are discussed on >> both of these pages: >> http://libguestfs.org/guestfs.3.html#environment-variables >> http://libguestfs.org/virt-v2v.1.html#environment-variables >> >> A note on the output mode >> ------------------------- >> >> You're going to be using '-o vdsm -os /storage_domain' and the other >> options: '--vdsm-image-uuid UUID', '--vdsm-vol-uuid UUID', >> '--vdsm-vm-uuid UUID' [http://libguestfs.org/virt-v2v.1.html#options] >> >> Implicit in the contract with '-o vdsm' is that VDSM is responsible >> for creating the output directories, running virt-v2v as user 36:36, >> and deleting the output directories if virt-v2v fails. > > VDSM is not only responsible for creating the output directories but also > the output files (we have to have it that way when the destination storage > domain resides in block-based device). We'll probably need to use a > technique that was used elsewhere, where output was streamed into those > files. so virt-v2v won't create the output files. why do we need to stream it via vdsm and not just give the path for v2v to write to?

> >> >> In this output mode virt-v2v doesn't create or clean up after itself >> or try to chown files (unlike '-o rhev'). >> >> HTH, >> >> Rich. >> >> -- >> Richard Jones, Virtualization Group, Red Hat >> http://people.redhat.com/~rjones >> Read my programming and virtualization blog: http://rwmj.wordpress.com >> virt-builder quickly builds VMs from scratch >> http://libguestfs.org/virt-builder.1.html >> > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >

On 10/02/2014 11:23 AM, Arik Hadas wrote: > > > ----- Original Message ----- >> On 09/30/2014 04:18 PM, Arik Hadas wrote: >>> >>> >>> ----- Original Message ----- >>>> To expand on the previous email, the web page is quite vague about >>>> what precise sources you want to import from. >>>> >>>> The ones supported by virt-v2v are: >>>> >>>> - VMware live vCenter: I assume you would want this. (Note ESXi is >>>> mentioned in the web page, but virt-v2v does not support import >>>> from ESXi directly) >>>> >>>> - OVA file: As Shahar wrote this bit, I'll assume you'll want to use >>>> it. >>>> >>>> - RHEL 5 Xen: SSH key management is going to be a problem here. >>>> >>>> - Local disk files: Seems unlikely you'll want this. >>>> >>>> - Citrix Xen, Hyper-V, etc.: Requires development work. >>>> >>>> - Physical machines >>>> >>>> VMware vCenter >>>> -------------- >>>> >>>> See: >>>> http://libguestfs.org/virt-v2v.1.html#input-from-vmware-vcenter-server >>>> >>>> The user will need to specify: >>>> - hostname of VMware vCenter server >>>> - optional port number >>>> - username >>>> - password >>>> - name of VMware Datacenter >>>> - hostname of VMware ESXi server [may be able to relax this in >>>> future] >>>> - whether or not to check HTTPS certs >>>> - name of the guest >>>> >>>> As detailed in the link above, there are various virsh commands you >>>> can run from VDSM to establish whether conversion is likely to work, >>>> and to get a list of guests which you can present to the user. >>>> >>>> OVA file >>>> -------- >>>> >>>> The user will need to specify: >>>> - OVA file (eg. location) >>>> >>>> I've no idea how the file will end up being accessible to VDSM. NFS >>>> mount? >>>> >>>> RHEL 5 Xen >>>> ---------- >>>> >>>> See: http://libguestfs.org/virt-v2v.1.html#input-from-rhel-5-xen >>>> >>>> I think we're going to have a problem with ssh keys, because this mode >>>> currently only works when the virt-v2v client can get passwordless >>>> access to the RHEL 5 Xen server using ssh-agent (note that Kerberos >>>> does not work). We might fix this in future, but at the moment you're >>>> somehow going to have to ensure that the oVirt node has a private key >>>> which is listed in the authorized_keys in the RHEL 5 Xen server. I >>>> simply cannot imagine a user interface that is going to make this >>>> usable. >>>> >>>> I should also say that few people care about RHEL 5 Xen imports. >>>> Citrix Xen imports OTOH, certainly, but those were out of scope for >>>> current upstream development for RHEL 7.1. >>>> >>>> Anyway, the user will have to specify: >>>> >>>> - hostname of RHEL 5 Xen server >>>> - username >>>> - password >>>> - name of the guest >>>> >>>> There are various virsh commands you can run from VDSM to establish >>>> whether conversion is likely to work, and to get a list of guests >>>> which you can present to the user. >>>> >>>> Physical machines >>>> ----------------- >>>> >>>> See also: http://libguestfs.org/virt-p2v.1.html >>>> [Update from previous email: I have now updated this page to contain >>>> the latest documentation.] >>>> >>>> Virt-v2v can import physical machines, but you have to boot the >>>> separate virt-p2v ISO on the physical machine, and generally speaking >>>> the process is directed from the physical machine. >>>> >>>> This makes it quite unlike the other input sources. (Users will still >>>> be able to use the self-boot + '-o rhev' method, ie. import into the >>>> Export Storage Domain, as with RHEL 6). >>>> >>>> If oVirt can (or does?) run a PXE server, then it's possible to serve a >>>> virt-p2v bootable image to physical machines, and it's also possible >>>> to automate it from the PXE server side. >>>> [http://libguestfs.org/virt-p2v.1.html#kernel-command-line-configuration] >>>> >>>> Environment variables >>>> --------------------- >>>> >>>> You may also want to set the following environment variables before >>>> running virt-v2v: >>>> >>>> LIBGUESTFS_BACKEND=direct >>>> >>>> This is currently required for VMware and RHEL 5 Xen conversions owing >>>> to a bug in libvirt (1134592) that was today pushed to RHEL 7.2. You >>>> should be able to drop this once libvirt is fixed. >>>> >>>> TMPDIR=... >>>> >>>> You might want to set this to control where large temporary files are >>>> created during the v2v process. >>>> >>>> Other environment variables that may affect virt-v2v are discussed on >>>> both of these pages: >>>> http://libguestfs.org/guestfs.3.html#environment-variables >>>> http://libguestfs.org/virt-v2v.1.html#environment-variables >>>> >>>> A note on the output mode >>>> ------------------------- >>>> >>>> You're going to be using '-o vdsm -os /storage_domain' and the other >>>> options: '--vdsm-image-uuid UUID', '--vdsm-vol-uuid UUID', >>>> '--vdsm-vm-uuid UUID' [http://libguestfs.org/virt-v2v.1.html#options] >>>> >>>> Implicit in the contract with '-o vdsm' is that VDSM is responsible >>>> for creating the output directories, running virt-v2v as user 36:36, >>>> and deleting the output directories if virt-v2v fails. >>> >>> VDSM is not only responsible for creating the output directories but also >>> the output files (we have to have it that way when the destination >>> storage >>> domain resides in block-based device). We'll probably need to use a >>> technique that was used elsewhere, where output was streamed into those >>> files. so virt-v2v won't create the output files. >> >> why do we need to stream it via vdsm and not just give the path for v2v >> to write to? > > actually we were talking about 2 possible solutions for block devices and I > mentioned only one of them. > the solutions we were talking about were: > > 1. vdsm creates the file, initially its size is 1G and the writing is made > via vdsm so when we reach ~80% the file is extended every time. > iiuc, that is the mechanism which is being used when we copy the images > from glance. that's the option I mentioned before. > > 2. vdsm creates the file and we gives the path for v2v to write to - in > this case we have to be sure that the file is big enough to contain the > data which is going to be written there. > > the problem with the second option is that the actual size of the converted > disks is unknown before the conversion. > > so the tradeoff is: > - in most of the cases the first option would probably be slower than the > second > - the second option will be based on estimations of the actual size of the > converted disks and since all we know is their virtual-sizes, we might > allocated much more space that we need so we'll need another phase of > shrinking the disks according to the amount of data that was actually > written - in this case the whole operation will be slower than it could be > - the first option is already implemented > > we still consider which option to choose.. can't you estimate with low overhead from v2v the expected size to pre-create at the right size?

also, this is only relevant to cow. if your target is raw for servers (which should be the default), you need to allocate everything to begin with, no extensions anyway. > >> >>> >>>> >>>> In this output mode virt-v2v doesn't create or clean up after itself >>>> or try to chown files (unlike '-o rhev'). >>>> >>>> HTH, >>>> >>>> Rich. >>>> >>>> -- >>>> Richard Jones, Virtualization Group, Red Hat >>>> http://people.redhat.com/~rjones >>>> Read my programming and virtualization blog: http://rwmj.wordpress.com >>>> virt-builder quickly builds VMs from scratch >>>> http://libguestfs.org/virt-builder.1.html >>>> >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >>

- the second option will be based on estimations of the actual size of the converted disks and since all we know is their virtual-sizes, we might allocated much more space that we need so we'll need another phase of shrinking the disks according to the amount of data that was actually written - in this case the whole operation will be slower than it could be