Backup solution using the API
--_000_196AB7D9F4C2F943BFC199940DF21D79016557467A0CKITMSX07kit_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGVsbG8sDQoNCm5vdyB0aGF0IG91ciBvVmlydCBDbHVzdGVyIHJ1bnMgZ3JlYXQsIEknZCBsaWtl IHRvIGtub3cgaWYgYW55Ym9keSBoYXMgZG9uZSBhIGJhY2t1cCBzb2x1dGlvbiBmb3Igb1ZpcnQg YmVmb3JlLg0KDQpNeSBiYXNpYyBpZGVhIGdvZXMgYXMgZm9sbG93czoNCi0gQ3JlYXRlIGEgSlNP TiBmaWxlIHdpdGggbWFjaGluZSdzIHByZWZlcmVuY2VzIGZvciBsYXRlciByZXN0b3JlLCBjcmVh dGUgYSBzbmFwc2hvdCwgc25hdGNoIHRoZSBkaXNrIGJ5IGl0cyBkaXNrLWlkIChjb3B5IGl0IHRv IHRoZSBmaWxlc2VydmVyKSwgdGhlbiBkZWxldGluZyB0aGUgc25hcHNob3QgYW5kIGJlIGRvbmUg d2l0aCBpdC4NCi0gT24gcmVzdG9yZSBJJ2QganVzdCBjcmVhdGUgYSBuZXcgVk0gd2l0aCB0aGUg RGlza3MgYW5kIE5JQ3Mgc3BlY2lmaWVkIGluIHRoZSBwcmVmZXJlbmNlczsgdGhlbiBJJ2QgZ28g YWhlYWQgYW5kIGFuZCBwdXQgYmFjayB0aGUgZGlzay4uLiB0DQoNCkkndmUgcGxheWVkIGEgbGl0 dGxlIGJpdCBhcm91bmQgd2l0aCBidWlsZGluZyBhIEpTT04gZmlsZSBhbmQgc28gZmFyIGl0IHdv cmtzIGdyZWF0OyBJIGhhdmVuJ3QgdHJpZWQgdG8gbWFrZSBhIFZNIHdpdGggdGhhdCwgdGhvdWdo Li4uDQoNClVzaW5nIHRoZSBleHBvcnQgZG9tYWluIG9yIGEgc2ltcGxlIGV4cG9ydCBjb21tYW5k IGlzIG5vdCB3aGF0IEkgd2FudCBzaW5jZSB5b3UnZCBoYXZlIHRvIHR1cm4gb2ZmIHRoZSBtYWNo aW5lIHRvIGRvIHRoYXQgLSBBRkFJSy4gQ29ycmVjdCBtZSBpZiB0aGF0IHNob3VsZCBub3QgYmUg dHJ1ZS4NCg0KTm93LCBiZWZvcmUgSSBnbyBpbnRvIGFueSBtb3JlIGhhc3NsZSwgaGFzIHNvbWVi b2R5IGVsc2Ugb2YgeW91IGRvbmUgYSBsaXZlLWJhY2t1cCBzb2x1dGlvbiBmb3Igb1ZpcnQ/IEFy ZSB0aGVyZSBhbnkgcmVjb21tZW5kYXRpb25zPyBUaGFua3MgZm9yIGFueSBoZWxwIHByb3ZpZGVk IQ0KDQpCZXN0IHJlZ2FyZHMNClRob21hcyBLZXBwbGVyDQo= --_000_196AB7D9F4C2F943BFC199940DF21D79016557467A0CKITMSX07kit_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUUkFOU0lUSU9OQUwv L0VOIj4KPEhUTUw+CjxIRUFEPgogIDxNRVRBIEhUVFAtRVFVSVY9IkNvbnRlbnQtVHlwZSIgQ09O VEVOVD0idGV4dC9odG1sOyBDSEFSU0VUPVVURi04Ij4KICA8TUVUQSBOQU1FPSJHRU5FUkFUT1Ii IENPTlRFTlQ9Ikd0a0hUTUwvNC42LjYiPgo8L0hFQUQ+CjxCT0RZPgpIZWxsbyw8QlI+CjxCUj4K bm93IHRoYXQgb3VyIG9WaXJ0IENsdXN0ZXIgcnVucyBncmVhdCwgSSdkIGxpa2UgdG8ga25vdyBp ZiBhbnlib2R5IGhhcyBkb25lIGEgYmFja3VwIHNvbHV0aW9uIGZvciBvVmlydCBiZWZvcmUuPEJS Pgo8QlI+Ck15IGJhc2ljIGlkZWEgZ29lcyBhcyBmb2xsb3dzOjxCUj4KLSBDcmVhdGUgYSBKU09O IGZpbGUgd2l0aCBtYWNoaW5lJ3MgcHJlZmVyZW5jZXMgZm9yIGxhdGVyIHJlc3RvcmUsIGNyZWF0 ZSBhIHNuYXBzaG90LCBzbmF0Y2ggdGhlIGRpc2sgYnkgaXRzIGRpc2staWQgKGNvcHkgaXQgdG8g dGhlIGZpbGVzZXJ2ZXIpLCB0aGVuIGRlbGV0aW5nIHRoZSBzbmFwc2hvdCBhbmQgYmUgZG9uZSB3 aXRoIGl0LjxCUj4KLSBPbiByZXN0b3JlIEknZCBqdXN0IGNyZWF0ZSBhIG5ldyBWTSB3aXRoIHRo ZSBEaXNrcyBhbmQgTklDcyBzcGVjaWZpZWQgaW4gdGhlIHByZWZlcmVuY2VzOyB0aGVuIEknZCBn byBhaGVhZCBhbmQgYW5kIHB1dCBiYWNrIHRoZSBkaXNrLi4uIHQ8QlI+CjxCUj4KSSd2ZSBwbGF5 ZWQgYSBsaXR0bGUgYml0IGFyb3VuZCB3aXRoIGJ1aWxkaW5nIGEgSlNPTiBmaWxlIGFuZCBzbyBm YXIgaXQgd29ya3MgZ3JlYXQ7IEkgaGF2ZW4ndCB0cmllZCB0byBtYWtlIGEgVk0gd2l0aCB0aGF0 LCB0aG91Z2guLi48QlI+CjxCUj4KVXNpbmcgdGhlIGV4cG9ydCBkb21haW4gb3IgYSBzaW1wbGUg ZXhwb3J0IGNvbW1hbmQgaXMgbm90IHdoYXQgSSB3YW50IHNpbmNlIHlvdSdkIGhhdmUgdG8gdHVy biBvZmYgdGhlIG1hY2hpbmUgdG8gZG8gdGhhdCAtIEFGQUlLLiBDb3JyZWN0IG1lIGlmIHRoYXQg c2hvdWxkIG5vdCBiZSB0cnVlLjxCUj4KPEJSPgpOb3csIGJlZm9yZSBJIGdvIGludG8gYW55IG1v cmUgaGFzc2xlLCBoYXMgc29tZWJvZHkgZWxzZSBvZiB5b3UgZG9uZSBhIGxpdmUtYmFja3VwIHNv bHV0aW9uIGZvciBvVmlydD8gQXJlIHRoZXJlIGFueSByZWNvbW1lbmRhdGlvbnM/IFRoYW5rcyBm b3IgYW55IGhlbHAgcHJvdmlkZWQhPEJSPgo8QlI+CkJlc3QgcmVnYXJkczxCUj4KVGhvbWFzIEtl cHBsZXIKPC9CT0RZPgo8L0hUTUw+CgA= --_000_196AB7D9F4C2F943BFC199940DF21D79016557467A0CKITMSX07kit_--
Hi, For the live-backup, i think you can make a live snapshot of the vm, and then clone a new vm from that snapshot, after that you can do export. 2014-11-27 23:12 GMT+08:00 Keppler, Thomas (PEBA) <thomas.keppler@kit.edu>:
Hello,
now that our oVirt Cluster runs great, I'd like to know if anybody has done a backup solution for oVirt before.
My basic idea goes as follows: - Create a JSON file with machine's preferences for later restore, create a snapshot, snatch the disk by its disk-id (copy it to the fileserver), then deleting the snapshot and be done with it. - On restore I'd just create a new VM with the Disks and NICs specified in the preferences; then I'd go ahead and and put back the disk... t
I've played a little bit around with building a JSON file and so far it works great; I haven't tried to make a VM with that, though...
Using the export domain or a simple export command is not what I want since you'd have to turn off the machine to do that - AFAIK. Correct me if that should not be true.
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
Best regards Thomas Keppler
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Thomas/plysan oVirt has the Backup/Restore api designated to provide the capabilities to backup/restore a vm. see here: http://www.ovirt.org/Features/Backup-Restore_API_Integration ----- Original Message -----
From: "plysan" <plysab@gmail.com> To: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu> Cc: users@ovirt.org Sent: Thursday, December 4, 2014 3:51:24 AM Subject: Re: [ovirt-users] Backup solution using the API
Hi,
For the live-backup, i think you can make a live snapshot of the vm, and then clone a new vm from that snapshot, after that you can do export.
2014-11-27 23:12 GMT+08:00 Keppler, Thomas (PEBA) < thomas.keppler@kit.edu > :
Hello,
now that our oVirt Cluster runs great, I'd like to know if anybody has done a backup solution for oVirt before.
My basic idea goes as follows: - Create a JSON file with machine's preferences for later restore, create a snapshot, snatch the disk by its disk-id (copy it to the fileserver), then deleting the snapshot and be done with it. - On restore I'd just create a new VM with the Disks and NICs specified in the preferences; then I'd go ahead and and put back the disk... t
I've played a little bit around with building a JSON file and so far it works great; I haven't tried to make a VM with that, though...
Using the export domain or a simple export command is not what I want since you'd have to turn off the machine to do that - AFAIK. Correct me if that should not be true.
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
Best regards Thomas Keppler
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------080407050206010406010603 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now? Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever) destroy the snapshot This would at least give some BMR capabilities of your VMs. Ideally, I'd also like to be able to create a snapshot from within the VM, do a yum update, see if I like it or not, if I do, then destroy the snap shot. If I don't, I want to promote the snapshot and boot from that, then destroy the "original". (Yes, I know, BTRFS is supposed to be the solution to this with in the VM at some point, isn't it?) I have looked at the oVirt backup API wiki, but the code examples look more like pseudo code than real world examples. --------------080407050206010406010603 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:<br> </div> <blockquote cite="mid:196AB7D9F4C2F943BFC199940DF21D79016557467A0C@KIT-MSX-07.kit.edu" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> <meta name="GENERATOR" content="GtkHTML/4.6.6"> <br> Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!<br> <br> </blockquote> <br> I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?<br> <br> Basically, I'd like to:<br> Tell VM to flush it's buffers<br> suspend VM<br> take snap shot of boot virtual disk<br> resume VM<br> backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever)<br> destroy the snapshot<br> <br> This would at least give some BMR capabilities of your VMs.<br> <br> Ideally, I'd also like to be able to create a snapshot from within the VM, do a yum update, see if I like it or not, if I do, then destroy the snap shot. If I don't, I want to promote the snapshot and boot from that, then destroy the "original".<br> (Yes, I know, BTRFS is supposed to be the solution to this with in the VM at some point, isn't it?)<br> <br> I have looked at the oVirt backup API wiki, but the code examples look more like pseudo code than real world examples.<br> <br> </body> </html> --------------080407050206010406010603--
On 12/10/2014 01:06 PM, Blaster wrote:
On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?
Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever) destroy the snapshot
This would at least give some BMR capabilities of your VMs.
Ideally, I'd also like to be able to create a snapshot from within the VM, do a yum update, see if I like it or not, if I do, then destroy the snap shot. If I don't, I want to promote the snapshot and boot from that, then destroy the "original". (Yes, I know, BTRFS is supposed to be the solution to this with in the VM at some point, isn't it?)
I have looked at the oVirt backup API wiki, but the code examples look more like pseudo code than real world examples.
that's "live merge". its in 3.5, but pending rhel/centos 7.1 (and accompanying 3.5.z patches around for bugs found during testing with that upcoming release)
----- Original Message -----
From: "Blaster" <Blaster@556nato.com> To: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, December 10, 2014 8:06:58 PM Subject: Re: [ovirt-users] Backup solution using the API
On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?
Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM
You don't need to suspend the vm, qemu can create live snapshot. When the snapshot is done, you don't care about future io, it will simply not included in the backup.
backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever)
You can by attaching the snapshot to another vm
destroy the snapshot
You can in ovirt 3.5.1 - we do not depend any more on future libvirt features. Adam, can you confirm?
This would at least give some BMR capabilities of your VMs.
Ideally, I'd also like to be able to create a snapshot from within the VM, do
You can do this "within" the vm by using the engine REST API. But this can be fragile - for example, if the vm pauses, your backup tool within the vm will never complete :-)
a yum update, see if I like it or not, if I do, then destroy the snap shot.
Possible in 3.5.1 using REST API
If I don't, I want to promote the snapshot and boot from that, then destroy the "original".
Same Nir
On 15/12/14 04:31 -0500, Nir Soffer wrote:
----- Original Message -----
From: "Blaster" <Blaster@556nato.com> To: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, December 10, 2014 8:06:58 PM Subject: Re: [ovirt-users] Backup solution using the API
On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?
Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM
You don't need to suspend the vm, qemu can create live snapshot. When the snapshot is done, you don't care about future io, it will simply not included in the backup.
backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever)
You can by attaching the snapshot to another vm
destroy the snapshot
You can in ovirt 3.5.1 - we do not depend any more on future libvirt features. Adam, can you confirm?
Yes, on 3.5.1 you'll be able to destroy the snapshot without impacting the running VM.
This would at least give some BMR capabilities of your VMs.
Ideally, I'd also like to be able to create a snapshot from within the VM, do
You can do this "within" the vm by using the engine REST API. But this can be fragile - for example, if the vm pauses, your backup tool within the vm will never complete :-)
a yum update, see if I like it or not, if I do, then destroy the snap shot.
Possible in 3.5.1 using REST API
If I don't, I want to promote the snapshot and boot from that, then destroy the "original".
Same
Nir
-- Adam Litke
Dear all, i am a little lost, i tried quite a few things with the snapshots, so far with python scripts I can iterate through the existing machines, take one, make a snapshot and all this. However, there are 2 problems I can not get around: 1. even when on 3.5.1 I can not delete a snapshot on a running VM, if I understood that correctly this relies on the Live Merge Feature where the code is available in vdsm already but it needs a certain libvirt version !?!? So question here is, can I delete a snapshot or not ? can I use only the rest API not python (excuse me I am not a developer) 2. when I attach a snapshot to another virtual machine, how do I do the backup then ? Does anybody have this already ? The environment is running on CentOS 7 (hypervisors), Centos 6 (hosted engine), the ovirt is on version 3.5.1, also we use gluster as a storage backend where the gluster servers are managed within the hosted engine in a separate cluster exporting the storage only. Regards Soeren -----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Adam Litke Sent: Tuesday, January 6, 2015 5:04 PM To: Nir Soffer Cc: Thomas Keppler (PEBA); users@ovirt.org Subject: Re: [ovirt-users] Backup solution using the API On 15/12/14 04:31 -0500, Nir Soffer wrote:
----- Original Message -----
From: "Blaster" <Blaster@556nato.com> To: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, December 10, 2014 8:06:58 PM Subject: Re: [ovirt-users] Backup solution using the API
On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?
Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM
You don't need to suspend the vm, qemu can create live snapshot. When the snapshot is done, you don't care about future io, it will simply not included in the backup.
backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever)
You can by attaching the snapshot to another vm
destroy the snapshot
You can in ovirt 3.5.1 - we do not depend any more on future libvirt features. Adam, can you confirm?
Yes, on 3.5.1 you'll be able to destroy the snapshot without impacting the running VM.
This would at least give some BMR capabilities of your VMs.
Ideally, I'd also like to be able to create a snapshot from within the VM, do
You can do this "within" the vm by using the engine REST API. But this can be fragile - for example, if the vm pauses, your backup tool within the vm will never complete :-)
a yum update, see if I like it or not, if I do, then destroy the snap shot.
Possible in 3.5.1 using REST API
If I don't, I want to promote the snapshot and boot from that, then destroy the "original".
Same
Nir
-- Adam Litke _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 11/02/15 17:30 +0000, Soeren Malchow wrote:
Dear all,
i am a little lost, i tried quite a few things with the snapshots, so far with python scripts I can iterate through the existing machines, take one, make a snapshot and all this.
However, there are 2 problems I can not get around:
1. even when on 3.5.1 I can not delete a snapshot on a running VM, if I understood that correctly this relies on the Live Merge Feature where the code is available in vdsm already but it needs a certain libvirt version !?!? So question here is, can I delete a snapshot or not ? can I use only the rest API not python (excuse me I am not a developer)
I can answer this one at least. For now, Centos 7 is lacking a new enough version of libvirt to support Live Merge (> 1.2.7). The virt-preview repository for Fedora is shipping a newer libvirt. You can check if vdsm will support live merge by executing the following on the host in question: sudo vdsClient -s 0 getVdsCaps | grep liveMerge RHEL 7.1 will be shipping a version with the neccessary backports to support live merge so I imagine Centos 7.1 will pick those up automatically? -- Adam Litke
Il 11/Feb/2015 19:10 "Adam Litke" <alitke@redhat.com> ha scritto:
On 11/02/15 17:30 +0000, Soeren Malchow wrote:
Dear all,
i am a little lost, i tried quite a few things with the snapshots, so far with python scripts I can iterate through the existing machines, take one, make a snapshot and all this.
However, there are 2 problems I can not get around:
1. even when on 3.5.1 I can not delete a snapshot on a running VM, if I understood that correctly this relies on the Live Merge Feature where the code is available in vdsm already but it needs a certain libvirt version !?!? So question here is, can I delete a snapshot or not ? can I use only the rest API not python (excuse me I am not a developer)
I can answer this one at least. For now, Centos 7 is lacking a new enough version of libvirt to support Live Merge (> 1.2.7). The virt-preview repository for Fedora is shipping a newer libvirt. You can check if vdsm will support live merge by executing the following on the host in question:
sudo vdsClient -s 0 getVdsCaps | grep liveMerge
RHEL 7.1 will be shipping a version with the neccessary backports to support live merge so I imagine Centos 7.1 will pick those up automatically?
-- Adam Litke _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I confirm that at least on f21 tc3 I was able to live delete snapshots, also into the active layer. See: http://lists.ovirt.org/pipermail/users/2014-November/029450.html and the whole thread. I hope that RHEL 7.1 (and CentOS) will have the merging into the active layer too.... and not only into the chain history. Gianluca
--_000_687766E0007010429FBFF98CA708248887423Fmcexch01mcongroup_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 RGVhciBhbGwsDQoNCnRoYW5rIHlvdSBmb3IgdGhlIGhlbHAsIHJpZ2h0IG5vdyB3ZSBhcmUgYXQg YSBzdGF0ZSB3aGVyZSB3ZSBjYW4gZG8gc25hcHNob3RzIGFuZCByZW1vdmUgdGhlbSwgc2luY2Ug dGhlcmUgYXJlIG5vIG92aXJ0IHBhY2thZ2VzIGZvciBGZWRvcmEgMjEgd2UgYXJlIHVzaW5nIGZl ZG9yYSAyMCArIHZpcnQtcHJldmlldy4NCg0KRXZlcnl0aGluZyBpcyB3b3JraW5nIGZpbmUgZnJv bSBhIHNwYW5zaG90IHBlcnNwZWN0aXZlLg0KDQpSZWdhcmRzDQpTb2VyZW4NCg0KRnJvbTogR2lh bmx1Y2EgQ2VjY2hpIFttYWlsdG86Z2lhbmx1Y2EuY2VjY2hpQGdtYWlsLmNvbV0NClNlbnQ6IFdl ZG5lc2RheSwgRmVicnVhcnkgMTEsIDIwMTUgMTA6MjMgUE0NClRvOiBBZGFtIExpdGtlDQpDYzog dXNlcnM7IFNvZXJlbiBNYWxjaG93OyBUaG9tYXMgS2VwcGxlciAoUEVCQSkNClN1YmplY3Q6IFJl OiBbb3ZpcnQtdXNlcnNdIEJhY2t1cCBzb2x1dGlvbiB1c2luZyB0aGUgQVBJDQoNCg0KSWwgMTEv RmViLzIwMTUgMTk6MTAgIkFkYW0gTGl0a2UiIDxhbGl0a2VAcmVkaGF0LmNvbTxtYWlsdG86YWxp dGtlQHJlZGhhdC5jb20+PiBoYSBzY3JpdHRvOg0KPg0KPiBPbiAxMS8wMi8xNSAxNzozMCArMDAw MCwgU29lcmVuIE1hbGNob3cgd3JvdGU6DQo+Pg0KPj4gRGVhciBhbGwsDQo+Pg0KPj4gaSBhbSBh IGxpdHRsZSBsb3N0LCBpIHRyaWVkIHF1aXRlIGEgZmV3IHRoaW5ncyB3aXRoIHRoZSBzbmFwc2hv dHMsIHNvDQo+PiBmYXIgd2l0aCBweXRob24gc2NyaXB0cyBJIGNhbiBpdGVyYXRlIHRocm91Z2gg dGhlIGV4aXN0aW5nIG1hY2hpbmVzLA0KPj4gdGFrZSBvbmUsIG1ha2UgYSBzbmFwc2hvdCBhbmQg YWxsIHRoaXMuDQo+Pg0KPj4gSG93ZXZlciwgdGhlcmUgYXJlIDIgcHJvYmxlbXMgSSBjYW4gbm90 IGdldCBhcm91bmQ6DQo+Pg0KPj4gMS4gZXZlbiB3aGVuIG9uIDMuNS4xIEkgY2FuIG5vdCBkZWxl dGUgYSBzbmFwc2hvdCBvbiBhIHJ1bm5pbmcgVk0sIGlmDQo+PiBJIHVuZGVyc3Rvb2QgdGhhdCBj b3JyZWN0bHkgdGhpcyByZWxpZXMgb24gdGhlIExpdmUgTWVyZ2UgRmVhdHVyZQ0KPj4gd2hlcmUg dGhlIGNvZGUgaXMgYXZhaWxhYmxlIGluIHZkc20gYWxyZWFkeSBidXQgaXQgbmVlZHMgYSBjZXJ0 YWluDQo+PiBsaWJ2aXJ0IHZlcnNpb24gIT8hPyAgU28gcXVlc3Rpb24gaGVyZSBpcywgY2FuIEkg ZGVsZXRlIGEgc25hcHNob3Qgb3INCj4+IG5vdCA/IGNhbiBJIHVzZSBvbmx5IHRoZSByZXN0IEFQ SSBub3QgcHl0aG9uICAoZXhjdXNlIG1lIEkgYW0gbm90IGENCj4+IGRldmVsb3BlcikNCj4NCj4N Cj4gSSBjYW4gYW5zd2VyIHRoaXMgb25lIGF0IGxlYXN0LiAgRm9yIG5vdywgQ2VudG9zIDcgaXMg bGFja2luZyBhIG5ldw0KPiBlbm91Z2ggdmVyc2lvbiBvZiBsaWJ2aXJ0IHRvIHN1cHBvcnQgTGl2 ZSBNZXJnZSAoPiAxLjIuNykuICBUaGUNCj4gdmlydC1wcmV2aWV3IHJlcG9zaXRvcnkgZm9yIEZl ZG9yYSBpcyBzaGlwcGluZyBhIG5ld2VyIGxpYnZpcnQuICBZb3UNCj4gY2FuIGNoZWNrIGlmIHZk c20gd2lsbCBzdXBwb3J0IGxpdmUgbWVyZ2UgYnkgZXhlY3V0aW5nIHRoZSBmb2xsb3dpbmcNCj4g b24gdGhlIGhvc3QgaW4gcXVlc3Rpb246DQo+DQo+ICAgc3VkbyB2ZHNDbGllbnQgLXMgMCBnZXRW ZHNDYXBzIHwgZ3JlcCBsaXZlTWVyZ2UNCj4NCj4gUkhFTCA3LjEgd2lsbCBiZSBzaGlwcGluZyBh IHZlcnNpb24gd2l0aCB0aGUgbmVjY2Vzc2FyeSBiYWNrcG9ydHMgdG8NCj4gc3VwcG9ydCBsaXZl IG1lcmdlIHNvIEkgaW1hZ2luZSBDZW50b3MgNy4xIHdpbGwgcGljayB0aG9zZSB1cA0KPiBhdXRv bWF0aWNhbGx5Pw0KPg0KPg0KPiAtLQ0KPiBBZGFtIExpdGtlDQo+IF9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IFVzZXJzIG1haWxpbmcgbGlzdA0KPiBV c2Vyc0BvdmlydC5vcmc8bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZz4NCj4gaHR0cDovL2xpc3RzLm92 aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzDQoNCkkgY29uZmlybSB0aGF0IGF0IGxlYXN0 IG9uIGYyMSB0YzMgSSB3YXMgYWJsZSB0byBsaXZlIGRlbGV0ZSBzbmFwc2hvdHMsICBhbHNvIGlu dG8gdGhlIGFjdGl2ZSBsYXllci4NClNlZToNCmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvcGlwZXJt YWlsL3VzZXJzLzIwMTQtTm92ZW1iZXIvMDI5NDUwLmh0bWwNCmFuZCB0aGUgd2hvbGUgdGhyZWFk Lg0KSSBob3BlIHRoYXQgUkhFTCA3LjEgKGFuZCBDZW50T1MpIHdpbGwgaGF2ZSB0aGUgbWVyZ2lu ZyBpbnRvIHRoZSBhY3RpdmUgbGF5ZXIgdG9vLi4uLiBhbmQgbm90IG9ubHkgaW50byB0aGUgY2hh aW4gaGlzdG9yeS4NCg0KR2lhbmx1Y2ENCg== --_000_687766E0007010429FBFF98CA708248887423Fmcexch01mcongroup_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6 IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy IDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWws IGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBpbjsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJ Zm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWlseToiVGltZXMgTmV3IFJvbWFuIixzZXJpZjt9 DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCglj b2xvcjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0KYTp2aXNpdGVkLCBzcGFu Lk1zb0h5cGVybGlua0ZvbGxvd2VkDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCgljb2xvcjpw dXJwbGU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpwDQoJe21zby1zdHlsZS1wcmlv cml0eTo5OTsNCgltc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzsNCgltYXJnaW4tcmlnaHQ6MGluOw0K CW1zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvOw0KCW1hcmdpbi1sZWZ0OjBpbjsNCglmb250LXNp emU6MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLHNlcmlmO30NCnNwYW4u RW1haWxTdHlsZTE4DQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFt aWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmOw0KCWNvbG9yOiMxRjQ5N0Q7fQ0KLk1zb0NocERlZmF1 bHQNCgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1mYW1pbHk6IkNhbGlicmki LHNhbnMtc2VyaWY7DQoJbXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVM7fQ0KQHBhZ2UgV29yZFNl Y3Rpb24xDQoJe3NpemU6OC41aW4gMTEuMGluOw0KCW1hcmdpbjo3MC44NXB0IDcwLjg1cHQgNTYu N3B0IDcwLjg1cHQ7fQ0KZGl2LldvcmRTZWN0aW9uMQ0KCXtwYWdlOldvcmRTZWN0aW9uMTt9DQot LT48L3N0eWxlPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPg0KPG86c2hhcGVkZWZhdWx0cyB2OmV4 dD0iZWRpdCIgc3BpZG1heD0iMTAyNiIgLz4NCjwveG1sPjwhW2VuZGlmXS0tPjwhLS1baWYgZ3Rl IG1zbyA5XT48eG1sPg0KPG86c2hhcGVsYXlvdXQgdjpleHQ9ImVkaXQiPg0KPG86aWRtYXAgdjpl eHQ9ImVkaXQiIGRhdGE9IjEiIC8+DQo8L286c2hhcGVsYXlvdXQ+PC94bWw+PCFbZW5kaWZdLS0+ DQo8L2hlYWQ+DQo8Ym9keSBsYW5nPSJERSIgbGluaz0iYmx1ZSIgdmxpbms9InB1cnBsZSI+DQo8 ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OyxzYW5z LXNlcmlmO2NvbG9yOiMxRjQ5N0Q7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkRlYXIgYWxs LDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxl PSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssc2Fucy1z ZXJpZjtjb2xvcjojMUY0OTdEO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNw OzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V UyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90 OyxzYW5zLXNlcmlmO2NvbG9yOiMxRjQ5N0Q7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPnRo YW5rIHlvdSBmb3IgdGhlIGhlbHAsIHJpZ2h0IG5vdyB3ZSBhcmUgYXQgYSBzdGF0ZSB3aGVyZSB3 ZSBjYW4gZG8gc25hcHNob3RzIGFuZCByZW1vdmUgdGhlbSwgc2luY2UgdGhlcmUgYXJlIG5vIG92 aXJ0DQogcGFja2FnZXMgZm9yIEZlZG9yYSAyMSB3ZSBhcmUgdXNpbmcgZmVkb3JhIDIwICYjNDM7 IHZpcnQtcHJldmlldy48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6 JnF1b3Q7Q2FsaWJyaSZxdW90OyxzYW5zLXNlcmlmO2NvbG9yOiMxRjQ5N0Q7bXNvLWZhcmVhc3Qt bGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250 LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzFGNDk3RDttc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+RXZlcnl0aGluZyBpcyB3b3JraW5nIGZpbmUgZnJvbSBh IHNwYW5zaG90IHBlcnNwZWN0aXZlLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250 LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzFGNDk3RDttc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEu MHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssc2Fucy1zZXJpZjtjb2xvcjojMUY0 OTdEO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5SZWdhcmRzPG86cD48L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250 LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssc2Fucy1zZXJpZjtj b2xvcjojMUY0OTdEO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5Tb2VyZW48bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YSBuYW1lPSJfTWFpbEVuZENvbXBv c2UiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWls eTomcXVvdDtDYWxpYnJpJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzFGNDk3RDttc28tZmFyZWFz dC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9hPjwvcD4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiPjxiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEx LjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LHNhbnMtc2VyaWYiPkZyb206PC9z cGFuPjwvYj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1m YW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OyxzYW5zLXNlcmlmIj4gR2lhbmx1Y2EgQ2VjY2hpIFtt YWlsdG86Z2lhbmx1Y2EuY2VjY2hpQGdtYWlsLmNvbV0NCjxicj4NCjxiPlNlbnQ6PC9iPiBXZWRu ZXNkYXksIEZlYnJ1YXJ5IDExLCAyMDE1IDEwOjIzIFBNPGJyPg0KPGI+VG86PC9iPiBBZGFtIExp dGtlPGJyPg0KPGI+Q2M6PC9iPiB1c2VyczsgU29lcmVuIE1hbGNob3c7IFRob21hcyBLZXBwbGVy IChQRUJBKTxicj4NCjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBCYWNrdXAgc29s dXRpb24gdXNpbmcgdGhlIEFQSTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHA+PGJyPg0KSWwgMTEvRmViLzIwMTUgMTk6 MTAgJnF1b3Q7QWRhbSBMaXRrZSZxdW90OyAmbHQ7PGEgaHJlZj0ibWFpbHRvOmFsaXRrZUByZWRo YXQuY29tIj5hbGl0a2VAcmVkaGF0LmNvbTwvYT4mZ3Q7IGhhIHNjcml0dG86PGJyPg0KJmd0Ozxi cj4NCiZndDsgT24gMTEvMDIvMTUgMTc6MzAgJiM0MzswMDAwLCBTb2VyZW4gTWFsY2hvdyB3cm90 ZTo8YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7IERlYXIgYWxsLDxicj4NCiZndDsmZ3Q7PGJy Pg0KJmd0OyZndDsgaSBhbSBhIGxpdHRsZSBsb3N0LCBpIHRyaWVkIHF1aXRlIGEgZmV3IHRoaW5n cyB3aXRoIHRoZSBzbmFwc2hvdHMsIHNvPGJyPg0KJmd0OyZndDsgZmFyIHdpdGggcHl0aG9uIHNj cmlwdHMgSSBjYW4gaXRlcmF0ZSB0aHJvdWdoIHRoZSBleGlzdGluZyBtYWNoaW5lcyw8YnI+DQom Z3Q7Jmd0OyB0YWtlIG9uZSwgbWFrZSBhIHNuYXBzaG90IGFuZCBhbGwgdGhpcy48YnI+DQomZ3Q7 Jmd0Ozxicj4NCiZndDsmZ3Q7IEhvd2V2ZXIsIHRoZXJlIGFyZSAyIHByb2JsZW1zIEkgY2FuIG5v dCBnZXQgYXJvdW5kOjxicj4NCiZndDsmZ3Q7PGJyPg0KJmd0OyZndDsgMS4gZXZlbiB3aGVuIG9u IDMuNS4xIEkgY2FuIG5vdCBkZWxldGUgYSBzbmFwc2hvdCBvbiBhIHJ1bm5pbmcgVk0sIGlmPGJy Pg0KJmd0OyZndDsgSSB1bmRlcnN0b29kIHRoYXQgY29ycmVjdGx5IHRoaXMgcmVsaWVzIG9uIHRo ZSBMaXZlIE1lcmdlIEZlYXR1cmU8YnI+DQomZ3Q7Jmd0OyB3aGVyZSB0aGUgY29kZSBpcyBhdmFp bGFibGUgaW4gdmRzbSBhbHJlYWR5IGJ1dCBpdCBuZWVkcyBhIGNlcnRhaW48YnI+DQomZ3Q7Jmd0 OyBsaWJ2aXJ0IHZlcnNpb24gIT8hPyZuYnNwOyBTbyBxdWVzdGlvbiBoZXJlIGlzLCBjYW4gSSBk ZWxldGUgYSBzbmFwc2hvdCBvcjxicj4NCiZndDsmZ3Q7IG5vdCA/IGNhbiBJIHVzZSBvbmx5IHRo ZSByZXN0IEFQSSBub3QgcHl0aG9uJm5ic3A7IChleGN1c2UgbWUgSSBhbSBub3QgYTxicj4NCiZn dDsmZ3Q7IGRldmVsb3Blcik8YnI+DQomZ3Q7PGJyPg0KJmd0Ozxicj4NCiZndDsgSSBjYW4gYW5z d2VyIHRoaXMgb25lIGF0IGxlYXN0LiZuYnNwOyBGb3Igbm93LCBDZW50b3MgNyBpcyBsYWNraW5n IGEgbmV3PGJyPg0KJmd0OyBlbm91Z2ggdmVyc2lvbiBvZiBsaWJ2aXJ0IHRvIHN1cHBvcnQgTGl2 ZSBNZXJnZSAoJmd0OyAxLjIuNykuJm5ic3A7IFRoZTxicj4NCiZndDsgdmlydC1wcmV2aWV3IHJl cG9zaXRvcnkgZm9yIEZlZG9yYSBpcyBzaGlwcGluZyBhIG5ld2VyIGxpYnZpcnQuJm5ic3A7IFlv dTxicj4NCiZndDsgY2FuIGNoZWNrIGlmIHZkc20gd2lsbCBzdXBwb3J0IGxpdmUgbWVyZ2UgYnkg ZXhlY3V0aW5nIHRoZSBmb2xsb3dpbmc8YnI+DQomZ3Q7IG9uIHRoZSBob3N0IGluIHF1ZXN0aW9u Ojxicj4NCiZndDs8YnI+DQomZ3Q7ICZuYnNwOyBzdWRvIHZkc0NsaWVudCAtcyAwIGdldFZkc0Nh cHMgfCBncmVwIGxpdmVNZXJnZTxicj4NCiZndDs8YnI+DQomZ3Q7IFJIRUwgNy4xIHdpbGwgYmUg c2hpcHBpbmcgYSB2ZXJzaW9uIHdpdGggdGhlIG5lY2Nlc3NhcnkgYmFja3BvcnRzIHRvPGJyPg0K Jmd0OyBzdXBwb3J0IGxpdmUgbWVyZ2Ugc28gSSBpbWFnaW5lIENlbnRvcyA3LjEgd2lsbCBwaWNr IHRob3NlIHVwPGJyPg0KJmd0OyBhdXRvbWF0aWNhbGx5Pzxicj4NCiZndDs8YnI+DQomZ3Q7PGJy Pg0KJmd0OyAtLSA8YnI+DQomZ3Q7IEFkYW0gTGl0a2U8YnI+DQomZ3Q7IF9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyPg0KJmd0OyBVc2VycyBtYWlsaW5n IGxpc3Q8YnI+DQomZ3Q7IDxhIGhyZWY9Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92 aXJ0Lm9yZzwvYT48YnI+DQomZ3Q7IDxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFp bG1hbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzPC9hPjxvOnA+PC9vOnA+PC9wPg0KPHA+SSBjb25maXJtIHRoYXQgYXQgbGVhc3Qg b24gZjIxIHRjMyBJIHdhcyBhYmxlIHRvIGxpdmUgZGVsZXRlIHNuYXBzaG90cywmbmJzcDsgYWxz byBpbnRvIHRoZSBhY3RpdmUgbGF5ZXIuPGJyPg0KU2VlOjxicj4NCjxhIGhyZWY9Imh0dHA6Ly9s aXN0cy5vdmlydC5vcmcvcGlwZXJtYWlsL3VzZXJzLzIwMTQtTm92ZW1iZXIvMDI5NDUwLmh0bWwi Pmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvcGlwZXJtYWlsL3VzZXJzLzIwMTQtTm92ZW1iZXIvMDI5 NDUwLmh0bWw8L2E+PGJyPg0KYW5kIHRoZSB3aG9sZSB0aHJlYWQuPGJyPg0KSSBob3BlIHRoYXQg UkhFTCA3LjEgKGFuZCBDZW50T1MpIHdpbGwgaGF2ZSB0aGUgbWVyZ2luZyBpbnRvIHRoZSBhY3Rp dmUgbGF5ZXIgdG9vLi4uLiBhbmQgbm90IG9ubHkgaW50byB0aGUgY2hhaW4gaGlzdG9yeS48bzpw PjwvbzpwPjwvcD4NCjxwPkdpYW5sdWNhPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvYm9keT4N CjwvaHRtbD4NCg== --_000_687766E0007010429FBFF98CA708248887423Fmcexch01mcongroup_--
----- Soeren Malchow <soeren.malchow@mcon.net> wrote:
Dear all,
thank you for the help, right now we are at a state where we can do snapshots and remove them, since there are no ovirt packages for Fedora 21 we are using fedora 20 + virt-preview.
Everything is working fine from a spanshot perspective.
Excellent. Glad to hear it!
Regards Soeren
From: Gianluca Cecchi [mailto:gianluca.cecchi@gmail.com] Sent: Wednesday, February 11, 2015 10:23 PM To: Adam Litke Cc: users; Soeren Malchow; Thomas Keppler (PEBA) Subject: Re: [ovirt-users] Backup solution using the API
Il 11/Feb/2015 19:10 "Adam Litke" <alitke@redhat.com<mailto:alitke@redhat.com>> ha scritto:
On 11/02/15 17:30 +0000, Soeren Malchow wrote:
Dear all,
i am a little lost, i tried quite a few things with the snapshots, so far with python scripts I can iterate through the existing machines, take one, make a snapshot and all this.
However, there are 2 problems I can not get around:
1. even when on 3.5.1 I can not delete a snapshot on a running VM, if I understood that correctly this relies on the Live Merge Feature where the code is available in vdsm already but it needs a certain libvirt version !?!? So question here is, can I delete a snapshot or not ? can I use only the rest API not python (excuse me I am not a developer)
I can answer this one at least. For now, Centos 7 is lacking a new enough version of libvirt to support Live Merge (> 1.2.7). The virt-preview repository for Fedora is shipping a newer libvirt. You can check if vdsm will support live merge by executing the following on the host in question:
sudo vdsClient -s 0 getVdsCaps | grep liveMerge
RHEL 7.1 will be shipping a version with the neccessary backports to support live merge so I imagine Centos 7.1 will pick those up automatically?
-- Adam Litke _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
I confirm that at least on f21 tc3 I was able to live delete snapshots, also into the active layer. See: http://lists.ovirt.org/pipermail/users/2014-November/029450.html and the whole thread. I hope that RHEL 7.1 (and CentOS) will have the merging into the active layer too.... and not only into the chain history.
Gianluca
----- Original Message -----
From: "Soeren Malchow" <soeren.malchow@mcon.net> To: "Adam Litke" <alitke@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Cc: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, February 11, 2015 7:30:48 PM Subject: Re: [ovirt-users] Backup solution using the API
Dear all,
i am a little lost, i tried quite a few things with the snapshots, so far with python scripts I can iterate through the existing machines, take one, make a snapshot and all this.
However, there are 2 problems I can not get around:
1. even when on 3.5.1 I can not delete a snapshot on a running VM, if I understood that correctly this relies on the Live Merge Feature where the code is available in vdsm already but it needs a certain libvirt version !?!? So question here is, can I delete a snapshot or not ? can I use only the rest API not python (excuse me I am not a developer)
2. when I attach a snapshot to another virtual machine, how do I do the backup then ? Does anybody have this already ? Hi Soeren, you can find detailed example for Backup/Restore flows here - http://www.ovirt.org/Features/Backup-Restore_API_Integration
thanks, laravot.
The environment is running on CentOS 7 (hypervisors), Centos 6 (hosted engine), the ovirt is on version 3.5.1, also we use gluster as a storage backend where the gluster servers are managed within the hosted engine in a separate cluster exporting the storage only.
Regards Soeren
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Adam Litke Sent: Tuesday, January 6, 2015 5:04 PM To: Nir Soffer Cc: Thomas Keppler (PEBA); users@ovirt.org Subject: Re: [ovirt-users] Backup solution using the API
On 15/12/14 04:31 -0500, Nir Soffer wrote:
----- Original Message -----
From: "Blaster" <Blaster@556nato.com> To: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, December 10, 2014 8:06:58 PM Subject: Re: [ovirt-users] Backup solution using the API
On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?
Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM
You don't need to suspend the vm, qemu can create live snapshot. When the snapshot is done, you don't care about future io, it will simply not included in the backup.
backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever)
You can by attaching the snapshot to another vm
destroy the snapshot
You can in ovirt 3.5.1 - we do not depend any more on future libvirt features. Adam, can you confirm?
Yes, on 3.5.1 you'll be able to destroy the snapshot without impacting the running VM.
This would at least give some BMR capabilities of your VMs.
Ideally, I'd also like to be able to create a snapshot from within the VM, do
You can do this "within" the vm by using the engine REST API. But this can be fragile - for example, if the vm pauses, your backup tool within the vm will never complete :-)
a yum update, see if I like it or not, if I do, then destroy the snap shot.
Possible in 3.5.1 using REST API
If I don't, I want to promote the snapshot and boot from that, then destroy the "original".
Same
Nir
-- Adam Litke _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Dear Liron, thanks alot, i know, i was following that, however the Live Merge Feature was missing in CentOS 7, that was already the discussion. Now on Fedora 20 the Live Merge works (we have to stick to Fedora then until this is in CentOS 7.1 ?! ) I am experimenting with scripts to iterate through all machines and do a backup for all machines on a separate VM, I will let you know how that goes Cheers Soeren -----Original Message----- From: Liron Aravot [mailto:laravot@redhat.com] Sent: Monday, February 16, 2015 10:15 AM To: Soeren Malchow Cc: Adam Litke; Nir Soffer; Thomas Keppler (PEBA); users@ovirt.org Subject: Re: [ovirt-users] Backup solution using the API ----- Original Message -----
From: "Soeren Malchow" <soeren.malchow@mcon.net> To: "Adam Litke" <alitke@redhat.com>, "Nir Soffer" <nsoffer@redhat.com> Cc: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, February 11, 2015 7:30:48 PM Subject: Re: [ovirt-users] Backup solution using the API
Dear all,
i am a little lost, i tried quite a few things with the snapshots, so far with python scripts I can iterate through the existing machines, take one, make a snapshot and all this.
However, there are 2 problems I can not get around:
1. even when on 3.5.1 I can not delete a snapshot on a running VM, if I understood that correctly this relies on the Live Merge Feature where the code is available in vdsm already but it needs a certain libvirt version !?!? So question here is, can I delete a snapshot or not ? can I use only the rest API not python (excuse me I am not a developer)
2. when I attach a snapshot to another virtual machine, how do I do the backup then ? Does anybody have this already ? Hi Soeren, you can find detailed example for Backup/Restore flows here - http://www.ovirt.org/Features/Backup-Restore_API_Integration
thanks, laravot.
The environment is running on CentOS 7 (hypervisors), Centos 6 (hosted engine), the ovirt is on version 3.5.1, also we use gluster as a storage backend where the gluster servers are managed within the hosted engine in a separate cluster exporting the storage only.
Regards Soeren
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Adam Litke Sent: Tuesday, January 6, 2015 5:04 PM To: Nir Soffer Cc: Thomas Keppler (PEBA); users@ovirt.org Subject: Re: [ovirt-users] Backup solution using the API
On 15/12/14 04:31 -0500, Nir Soffer wrote:
----- Original Message -----
From: "Blaster" <Blaster@556nato.com> To: "Thomas Keppler (PEBA)" <thomas.keppler@kit.edu>, users@ovirt.org Sent: Wednesday, December 10, 2014 8:06:58 PM Subject: Re: [ovirt-users] Backup solution using the API
On 11/27/2014 9:12 AM, Keppler, Thomas (PEBA) wrote:
Now, before I go into any more hassle, has somebody else of you done a live-backup solution for oVirt? Are there any recommendations? Thanks for any help provided!
I've been looking for a similar scheme for the last year. It was not (really) possible in the past as there wasn't any way to destroy a snapshot w/o shutting down the VM. Is this still the case, or are snap shots fully implemented now?
Basically, I'd like to: Tell VM to flush it's buffers suspend VM take snap shot of boot virtual disk resume VM
You don't need to suspend the vm, qemu can create live snapshot. When the snapshot is done, you don't care about future io, it will simply not included in the backup.
backup the virtual boot disk from the Hypervisor using standard commands (tar, cp, whatever)
You can by attaching the snapshot to another vm
destroy the snapshot
You can in ovirt 3.5.1 - we do not depend any more on future libvirt features. Adam, can you confirm?
Yes, on 3.5.1 you'll be able to destroy the snapshot without impacting the running VM.
This would at least give some BMR capabilities of your VMs.
Ideally, I'd also like to be able to create a snapshot from within the VM, do
You can do this "within" the vm by using the engine REST API. But this can be fragile - for example, if the vm pauses, your backup tool within the vm will never complete :-)
a yum update, see if I like it or not, if I do, then destroy the snap shot.
Possible in 3.5.1 using REST API
If I don't, I want to promote the snapshot and boot from that, then destroy the "original".
Same
Nir
-- Adam Litke _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (9)
-
Adam Litke -
Blaster -
Gianluca Cecchi -
Itamar Heim -
Keppler, Thomas (PEBA) -
Liron Aravot -
Nir Soffer -
plysan -
Soeren Malchow