<div dir="ltr"><div>Adam,</div><div>Just out of curiosity: when you write "v2v has promised" - what exactly do you mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I that implemented the integration with virt-v2v? I'm not aware of such a promise by any of these options :)</div><div><br></div><div>Anyway, let's say that you were given such a promise by someone and thus consider that mechanism to be deprecated - it doesn't really matter.</div><div>The current implementation doesn't well fit to this flow (it requires per-volume job, it creates leases that are not needed for template's disks, ...) and with the "next-gen API" with proper support for virt flows not even being discussed with us (and iiuc also not with the infra team) yet, I don't understand what do you suggest except for some strong, though irrelevant, statements.</div><div>I suggest loud and clear to reuse (not to add dependencies, not to enhance, ..) an existing mechanism for a very similar flow of virt-v2v that works well and simple.</div><div><br></div><div>Do you "promise" to implement your "next gen API" for 4.1 as an alternative?</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 6, 2016 at 5:04 PM, Adam Litke <span dir="ltr"><<a href="mailto:alitke@redhat.com" target="_blank">alitke@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 05/12/16 11:17 +0200, Arik Hadas wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
On Mon, Dec 5, 2016 at 10:05 AM, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>> wrote:<br>
<br>
On Sun, Dec 4, 2016 at 8:50 PM, Shmuel Melamud <<a href="mailto:smelamud@redhat.com" target="_blank">smelamud@redhat.com</a>> wrote:<br>
><br>
> Hi!<br>
><br>
> I'm currently working on integration of virt-sysprep into oVirt.<br>
><br>
> Usually, if user creates a template from a regular VM, and then creates<br>
new VMs from this template, these new VMs inherit all configuration of the<br>
original VM, including SSH keys, UDEV rules, MAC addresses, system ID,<br>
hostname etc. It is unfortunate, because you cannot have two network<br>
devices with the same MAC address in the same network, for example.<br>
><br>
> To avoid this, user must clean all machine-specific configuration from<br>
the original VM before creating a template from it. You can do this<br>
manually, but there is virt-sysprep utility that does this automatically.<br>
><br>
> Ideally, virt-sysprep should be seamlessly integrated into template<br>
creation process. But the first step is to create a simple button: user<br>
selects a VM, clicks the button and oVirt executes virt-sysprep on the VM.<br>
><br>
> virt-sysprep works directly on VM's filesystem. It accepts list of all<br>
disks of the VM as parameters:<br>
><br>
> virt-sysprep -a disk1.img -a disk2.img -a disk3.img<br>
><br>
> The architecture is as follows: command on the Engine side runs a job on<br>
VDSM side and tracks its success/failure. The job on VDSM side runs<br>
virt-sysprep.<br>
><br>
> The question is how to implement the job correctly?<br>
><br>
> I thought about using storage jobs, but they are designed to work only<br>
with a single volume, correct?<br>
<br>
New storage verbs are volume based. This make it easy to manage<br>
them on the engine side, and will allow parallelizing volume operations<br>
on single or multiple hosts.<br>
<br>
A storage volume job is using sanlock lease on the modified volume<br>
and volume generation number. If a host running pending jobs becomes<br>
non-responsive and cannot be fenced, we can detect the state of<br>
the job, fence the job, and start the job on another host.<br>
<br>
In the SPM task, if a host becomes non-responsive and cannot be<br>
fenced, the whole setup is stuck, there is no way to perform any<br>
storage operation.<br>
> Is is possible to use them with operation that is performed on multiple<br>
volumes?<br>
> Or, alternatively, is it possible to use some kind of 'VM jobs' - that<br>
work on VM at whole?<br>
<br>
We can do:<br>
<br>
1. Add jobs with multiple volumes leases - can make error handling very<br>
complex. How do tell a job state if you have multiple leases? which<br>
volume generation you use?<br>
<br>
2. Use volume job using one of the volumes (the boot volume?). This does<br>
not protect the other volumes from modification but engine is<br>
responsible<br>
for this.<br>
<br>
3. Use new "vm jobs", using a vm lease (should be available this week<br>
on master).<br>
This protects a vm during sysprep from starting the vm.<br>
We still need a generation to detect the job state, I think we can<br>
use the sanlock<br>
lease generation for this.<br>
<br>
I like the last option since sysprep is much like running a vm.<br>
> How v2v solves this problem?<br>
<br>
It does not.<br>
<br>
v2v predates storage volume jobs. It does not use volume leases and<br>
generation<br>
and does have any way to recover if a host running v2v becomes<br>
non-responsive<br>
and cannot be fenced.<br>
<br>
It also does not use the jobs framework and does not use a thread pool for<br>
v2v jobs, so it has no limit on the number of storage operations on a host.<br>
<br>
<br>
Right, but let's be fair and present the benefits of v2v-jobs as well:<br>
1. it is the simplest "infrastructure" in terms of LOC<br>
</blockquote>
<br></div></div>
It is also deprecated. V2V has promised to adopt the richer Host Jobs<br>
API in the future.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. it is the most efficient mechanism in terms of interactions between the<br>
engine and VDSM (it doesn't require new verbs/call, the data is attached to<br>
VdsStats; probably the easiest mechanism to convert to events)<br>
</blockquote>
<br></span>
Engine is already polling the host jobs API so I am not sure I agree<br>
with you here.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. it is the most efficient implementation in terms of interaction with the<br>
database (no date is persisted into the database, no polling is done)<br>
</blockquote>
<br></span>
Again, we're already using the Host Jobs API. We'll gain efficiency<br>
by migrating away from the old v2v API and having a single, unified<br>
approach (Host Jobs).<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Currently we have 3 mechanisms to report jobs:<br>
1. VM jobs - that is currently used for live-merge. This requires the VM entity<br>
to exist in VDSM, thus not suitable for virt-sysprep.<br>
</blockquote>
<br></span>
Correct, not appropriate for this application.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. storage jobs - complicated infrastructure, targeted for recovering from<br>
failures to maintain storage consistency. Many of the things this<br>
infrastructure knows to handle is irrelevant for virt-sysprep flow, and the<br>
fact that virt-sysprep is invoked on VM rather than particular disk makes it<br>
less suitable.<br>
</blockquote>
<br></span>
These are more appropriately called HostJobs and the have the<br>
following semantics:<br>
- They represent an external process running on a single host<br>
- They are not persisted. If the host or vdsm restarts, the job is<br>
aborted<br>
- They operate on entities. Currently storage is the first adopter<br>
of the infrastructure but virt was going to adopt these for the<br>
next-gen API. Entities can be volumes, storage domains, vms,<br>
network interfaces, etc.<br>
- Job status and progress is reported by the Host Jobs API. If a job<br>
is not present, then the underlying entitie(s) must be polled by<br>
engine to determine the actual state.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. V2V jobs - no mechanism is provided to resume failed jobs, no leases, etc<br>
</blockquote>
<br></span>
This is the old infra upon which Host Jobs are built. v2v has<br>
promised to move to Host Jobs in the future so we should not add new<br>
dependencies to this code.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have some arguments for using V2V-like jobs [1]:<br>
1. creating template from vm is rarely done - if host goes unresponsive or any<br>
other failure is detected we can just remove the template and report the error<br>
</blockquote>
<br></span>
We can chose this error handling with Host Jobs as well.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. the phase of virt-sysprep is, unlike typical storage operation, short -<br>
reducing the risk of failures during the process <br>
</blockquote>
<br></span>
Reduced risk of failures is never an excuse to have lax error<br>
handling. The storage flavored host jobs provide tons of utilities<br>
for making error handling standardized, easy to implement, and<br>
correct.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. during the operation the VM is down - by locking the VM/template and its<br>
disks on the engine side, we render leases-like mechanism redundant<br>
</blockquote>
<br></span>
Eventually we want to protect all operations on storage with sanlock<br>
leases. This is safer and allows for a more distributed approach to<br>
management. Again, the use of leases correctly in host jobs requires<br>
about 5 lines of code. The benefits of standardization far outweigh<br>
any perceived simplification resulting from omitting it.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
4. in the worst case - the disk will not be corrupted (only some of the data<br>
might be removed).<br>
</blockquote>
<br></span>
Again, the way engine chooses to handle job failures is independent of<br>
the mechanism. Let's separate that from this discussion.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So I think that the mechanism for storage jobs is an over-kill for this case.<br>
We can keep it simple by generalise the V2V-job for other virt-tools jobs, like<br>
virt-sysprep.<br>
</blockquote>
<br></span>
I think we ought to standardize on the Host Jobs framework where we<br>
can collaborate on unit tests, standardized locking and error<br>
handling, abort logic, etc. When v2v moves to host jobs then we will<br>
have a unified method of handling ephemeral jobs that are tied to<br>
entities.<span class="HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Adam Litke<br>
</font></span></blockquote></div><br></div>