[Users] VM Migration failed

Edgars M. edgars.mazurs at gmail.com
Mon Jan 20 10:38:44 UTC 2014


Ok, so I shut down VM, unchecked Attach CD, powered on VM and now migration
succeeded. Go figure.

Thanks everyone for your help! I still dont know what is the cause of this
issue, but at least I know how to fix it.

Edgars


On Mon, Jan 20, 2014 at 11:26 AM, Edgars M. <edgars.mazurs at gmail.com> wrote:

> No, I did not that, but I did Eject under Change CD.
>
> Edgars
>
>
> On Mon, Jan 20, 2014 at 11:24 AM, Dafna Ron <dron at redhat.com> wrote:
>
>> so if you selected it as first boot device, after you installed did you
>> shut down the vm and remove the vm from the boot?
>>
>>
>> On 01/20/2014 10:21 AM, Edgars M. wrote:
>>
>>> Hi
>>>
>>> I specified ISO image and chose CD-ROM as first boot device.
>>>
>>> Edgars
>>>
>>>
>>> On Mon, Jan 20, 2014 at 11:19 AM, Dafna Ron <dron at redhat.com <mailto:
>>> dron at redhat.com>> wrote:
>>>
>>>     when you booted the vm's for the first time, did you specify the
>>>     ISO in vm's boot sequence or did you use run-once?
>>>
>>>
>>>     On 01/20/2014 09:53 AM, Edgars M. wrote:
>>>
>>>         Hi again
>>>
>>>         I tried to reproduce this issue and unfortunately I couldn't.
>>>         So there must be something else, but I cannot figure out what
>>>         is wrong, because it is happening only with some VMs.
>>>         Nonetheless I attached new log files to investigate.
>>>
>>>         And here is some more info:
>>>         ovirt-engine-setup-3.3.2-1.el6.noarch
>>>         ovirt-engine-webadmin-portal-3.3.2-1.el6.noarch
>>>         ovirt-host-deploy-java-1.1.2-1.el6.noarch
>>>         ovirt-engine-restapi-3.3.2-1.el6.noarch
>>>         ovirt-image-uploader-3.3.1-1.el6.noarch
>>>         ovirt-release-el6-10-1.noarch
>>>         ovirt-engine-userportal-3.3.2-1.el6.noarch
>>>         ovirt-engine-backend-3.3.2-1.el6.noarch
>>>         ovirt-engine-websocket-proxy-3.3.2-1.el6.noarch
>>>         ovirt-engine-dbscripts-3.3.2-1.el6.noarch
>>>         ovirt-log-collector-3.3.1-1.el6.noarch
>>>         ovirt-engine-sdk-python-3.3.0.8-1.el6.noarch
>>>         ovirt-engine-cli-3.3.0.6-1.el6.noarch
>>>         ovirt-engine-tools-3.3.2-1.el6.noarch
>>>         ovirt-iso-uploader-3.3.1-1.el6.noarch
>>>         ovirt-engine-lib-3.3.2-1.el6.noarch
>>>         ovirt-engine-3.3.2-1.el6.noarch
>>>         ovirt-host-deploy-1.1.2-1.el6.noarch
>>>
>>>         vdsm-xmlrpc-4.13.2-1.el6.noarch
>>>         vdsm-python-4.13.2-1.el6.x86_64
>>>         vdsm-python-cpopen-4.13.2-1.el6.x86_64
>>>         vdsm-4.13.2-1.el6.x86_64
>>>         vdsm-cli-4.13.2-1.el6.noarch
>>>
>>>         libvirt-lock-sanlock-0.10.2-29.el6_5.2.x86_64
>>>         libvirt-client-0.10.2-29.el6_5.2.x86_64
>>>         libvirt-python-0.10.2-29.el6_5.2.x86_64
>>>         libvirt-0.10.2-29.el6_5.2.x86_64
>>>
>>>         CentOS 6.5 x86_64 on both Engine and Nodes.
>>>
>>>         SELinux disabled everywhere (I know, I know).
>>>         There is no any snapshots for these VMs. These VMs were
>>>         installed by booting from ISO image.
>>>         There is nothing in libvirtd.log files
>>>
>>>
>>>         BR
>>>         Edgars
>>>
>>>
>>>
>>>         On Sat, Jan 18, 2014 at 6:39 PM, Dafna Ron <dron at redhat.com
>>>         <mailto:dron at redhat.com> <mailto:dron at redhat.com
>>>         <mailto:dron at redhat.com>>> wrote:
>>>
>>>             sounds like this is the issue to me... Edgars can you try to
>>>             confirm that? :)
>>>             Michal, was there a bug opened for this? I think that
>>>         perhaps we
>>>             should also add a clear error message - it would help
>>>         debug this
>>>             more easily.
>>>
>>>
>>>
>>>             On 01/18/2014 05:34 PM, Michal Skrivanek wrote:
>>>
>>>
>>>                     On 18 Jan 2014, at 18:06, Itamar Heim
>>>         <iheim at redhat.com <mailto:iheim at redhat.com>
>>>                     <mailto:iheim at redhat.com
>>>
>>>         <mailto:iheim at redhat.com>>> wrote:
>>>
>>>                         On 01/18/2014 03:44 PM, Edgars M. wrote:
>>>                         Hi
>>>
>>>                         Thanks for your help. I will provide all log
>>>         files a
>>>                         little bit later,
>>>                         but so far I have noticed some pattern when
>>>         migration
>>>                         fails. Those
>>>                         particular VMs, which fails to migrate, were
>>>         installed
>>>                         from ISO image,
>>>                         usual installation. So, I believe you can
>>>         reproduce
>>>                         the issue by this:
>>>
>>>                         1. Upload ISO image to ISO domain
>>>                         2. Install new OS by booting from ISO image
>>>         and check
>>>                         Attach CD in Boot
>>>                         Options
>>>                         3. Delete ISO image from ISO domain
>>>                         4. Try to migrate VM to another host.
>>>
>>>
>>>                         I have not tried this yet, but I have noticed
>>> that
>>>                         only those VMs fails
>>>                         which had been install from ISO image which is
>>>         not in
>>>                         ISO domain
>>>                         anymore. I have also VMs created from
>>>         templates and
>>>                         those VMs I can
>>>                         migrate just fine.
>>>
>>>                     did you stop the VMs post install? did you try to
>>>         start
>>>                     them without the iso attached?
>>>                     otherwise, you can't start them, as there is a
>>>         missing iso
>>>                     for the target qemu process
>>>
>>>                 Indeed. That is a known libvirt/qemu issue that even
>>>         though
>>>                 the CD is defined as optional and VM can be started
>>>         without it
>>>                 on original host, it fails upon migration when
>>>         destination is
>>>                 being created.
>>>
>>>                 Thanks,
>>>                 michal
>>>
>>>                         I will provide more log files later.
>>>
>>>                         BR
>>>                         Edgars
>>>
>>>
>>>                         On Sat, Jan 18, 2014 at 1:05 PM, Dafna Ron
>>>                         <dron at redhat.com <mailto:dron at redhat.com>
>>>         <mailto:dron at redhat.com <mailto:dron at redhat.com>>
>>>                         <mailto:dron at redhat.com
>>>         <mailto:dron at redhat.com> <mailto:dron at redhat.com
>>>         <mailto:dron at redhat.com>>>> wrote:
>>>
>>>                             I looked at the logs log and only vdsm
>>>         from node2
>>>                         (which appears to
>>>                             be the src node) seems to have full info.
>>>
>>>                             Please attach the following complete logs:
>>>                             vdsm log from dst node
>>>                             engine log
>>>                             libvirt logs from both nodes.
>>>
>>>                             Also, can you please answer the following?
>>>                             libvirt and vdsm you are using?
>>>                             what is selinux status on both hosts
>>>                         (enforcing/permissive)?
>>>                             do you have snapshots on the vm or does it
>>>         happen
>>>                         on a newly created
>>>                             disk?
>>>                             when you create the disk, is it from a
>>>         template or
>>>                         is it a new image?
>>>
>>>                             Thanks,
>>>
>>>                             Dafna
>>>
>>>
>>>
>>>
>>>                             On 01/17/2014 10:21 PM, Itamar Heim wrote:
>>>
>>>                                 On 01/18/2014 12:07 AM, Meital
>>>         Bourvine wrote:
>>>
>>>                                     I opened a bug about missing info
>>>         in UI:
>>>         https://bugzilla.redhat.com/__show_bug.cgi?id=1054994
>>>                                                           <
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1054994>
>>>
>>>                                     It actually failed with this error:
>>>         Thread-1417::DEBUG::2014-01-17
>>>         17:01:28,344::vm::768::vm.Vm::__(run)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::starting
>>>                                     migration monitor thread
>>>         Thread-1415::DEBUG::2014-01-17
>>>          17:01:28,409::__libvirtconnection::108::__
>>> libvirtconnection::(wrapper)
>>>                                     Unknown libvirterror: ecode: 38
>>>         edom: 42
>>>                         level: 2 message:
>>>                                     Failed to inquire lock: No such
>>>         process
>>>         Thread-1415::DEBUG::2014-01-17
>>>         17:01:28,409::vm::745::vm.Vm::__(cancel)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::canceling
>>>                                     migration downtime thread
>>>         Thread-1415::DEBUG::2014-01-17
>>>         17:01:28,409::vm::815::vm.Vm::__(stop)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::stopping
>>>                                     migration monitor thread
>>>         Thread-1416::DEBUG::2014-01-17
>>>         17:01:28,409::vm::742::vm.Vm::__(run)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::migration
>>>                                     downtime thread exiting
>>>         Thread-1415::ERROR::2014-01-17
>>>         17:01:28,410::vm::238::vm.Vm::__(_recover)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::Failed to
>>>                                     inquire lock: No such process
>>>         Thread-1415::ERROR::2014-01-17
>>>         17:01:28,619::vm::337::vm.Vm::__(run)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::Failed
>>>                         to migrate
>>>                                     Traceback (most recent call last):
>>>                                         File "/usr/share/vdsm/vm.py",
>>> line
>>>                         323, in run
>>>                                                  self.___
>>> startUnderlyingMigration()
>>>                                         File "/usr/share/vdsm/vm.py",
>>>         line 400, in
>>>                                     _startUnderlyingMigration
>>>                                           None, maxBandwidth)
>>>                                         File "/usr/share/vdsm/vm.py",
>>> line
>>>                         838, in f
>>>                                           ret = attr(*args, **kwargs)
>>>                                         File
>>>          "/usr/lib64/python2.6/site-__packages/vdsm/__
>>> libvirtconnection.py",
>>>                                     line 76, in wrapper
>>>                                           ret = f(*args, **kwargs)
>>>                                         File
>>>                         "/usr/lib64/python2.6/site-__
>>> packages/libvirt.py",
>>>                                     line 1178, in migrateToURI2
>>>                                           if ret == -1: raise
>>> libvirtError
>>>                                     ('virDomainMigrateToURI2()
>>>         failed', dom=self)
>>>                                     libvirtError: Failed to inquire
>>>         lock: No
>>>                         such process
>>>         Thread-26::ERROR::2014-01-17
>>>          17:01:28,917::sampling::355::__vm.Vm::(collect)
>>>          vmId=`b8787906-187a-4234-a0c9-__58fc4ddf2a57`::Stats
>>>                                     function failed:
>>>         <AdvancedStatsFunction
>>>                         _highWrite at 0x26efb58>
>>>
>>>                                     The problem is that it doesn't say
>>>         which
>>>                         process...
>>>
>>>
>>>                                 this looks like noise post the migration
>>>                         failing, with probably
>>>                                 a more relevant error in libvirt log
>>>
>>>
>>>                                     ----- Original Message -----
>>>
>>>                                         From: "Itamar Heim"
>>>         <iheim at redhat.com <mailto:iheim at redhat.com>
>>>                         <mailto:iheim at redhat.com
>>>         <mailto:iheim at redhat.com>>
>>>                                         <mailto:iheim at redhat.com
>>>         <mailto:iheim at redhat.com>
>>>
>>>                         <mailto:iheim at redhat.com
>>>         <mailto:iheim at redhat.com>>>>
>>>                                         To: "Edgars M."
>>>                         <edgars.mazurs at gmail.com
>>>         <mailto:edgars.mazurs at gmail.com>
>>>         <mailto:edgars.mazurs at gmail.com <mailto:edgars.mazurs at gmail.com
>>> >>
>>>                                                 <mailto:
>>> edgars.mazurs at gmail.com <mailto:edgars.mazurs at gmail.com>
>>>
>>>                         <mailto:edgars.mazurs at gmail.com
>>>         <mailto:edgars.mazurs at gmail.com>>>>, "Meital Bourvine"
>>>                                         <mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>
>>>                         <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>>
>>>                         <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com> <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>>>>
>>>                                         Cc: users at ovirt.org
>>>         <mailto:users at ovirt.org>
>>>                         <mailto:users at ovirt.org
>>>         <mailto:users at ovirt.org>> <mailto:users at ovirt.org
>>>         <mailto:users at ovirt.org>
>>>
>>>                         <mailto:users at ovirt.org
>>>
>>>         <mailto:users at ovirt.org>>>, "Michal
>>>                                         Skrivanek"
>>>         <mskrivan at redhat.com <mailto:mskrivan at redhat.com>
>>>                         <mailto:mskrivan at redhat.com
>>>         <mailto:mskrivan at redhat.com>>
>>>                                         <mailto:mskrivan at redhat.com
>>>         <mailto:mskrivan at redhat.com>
>>>
>>>                         <mailto:mskrivan at redhat.com
>>>         <mailto:mskrivan at redhat.com>>>>
>>>                                         Sent: Friday, January 17, 2014
>>>         9:47:11 PM
>>>                                         Subject: Re: [Users] VM
>>>         Migration failed
>>>
>>>                                         On 01/17/2014 06:25 PM, Edgars
>>>         M. wrote:
>>>
>>>                                             Hi Meital
>>>
>>>                                             I tried to migrate another
>>>         VM and
>>>                         it also failed.
>>>
>>>                                             This is what I get in UI:
>>>
>>>
>>>                                             Migration started (VM:
>>>         nophpapp01,
>>>                         Source:
>>>                                             novmnode1, Destination:
>>>                                             novmnode2, User: edgarsm).
>>>                                             Migration failed due to
>>> Error:
>>>                         Fatal error during
>>>                                             migration. Trying to
>>>                                             migrate to another Host (VM:
>>>                         nophpapp01, Source:
>>>                                             novmnode1, Destination:
>>>                                             novmnode2).
>>>                                             Migration failed due to
>>> Error:
>>>                         Fatal error during
>>>                                             migration (VM:
>>>                                             nophpapp01, Source:
>>> novmnode1,
>>>                         Destination: novmnode2).
>>>
>>>                                             There is nothing in
>>>                         /var/log/messages, neither on
>>>                                             engine server nor nodes.
>>>
>>>
>>>                                             See attachments for engine
>>> and
>>>                         vdsm logs. (I am
>>>                                             migrating from vmnode1
>>>                                             to vmnode2)
>>>
>>>
>>>                                         whatever the issue is here,
>>>         can you
>>>                         please open a bug on
>>>                                         trying to
>>>                                         return more info to user on the
>>>                         migration error itself
>>>                                         (for easier
>>>                                         troubleshooting).
>>>
>>>                                         thanks,
>>>                                               Itamar
>>>
>>>
>>>                                             Thanks for the help
>>>                                             Edgars
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>                                             On Fri, Jan 17, 2014 at
>>>         4:46 PM,
>>>                         Meital Bourvine
>>>                                             <mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>
>>>                         <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>>
>>>                         <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com> <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>>>
>>>                                                    <mailto:
>>> mbourvin at redhat.com <mailto:mbourvin at redhat.com>
>>>                         <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>>
>>>                                                    <mailto:
>>> mbourvin at redhat.com <mailto:mbourvin at redhat.com>
>>>                         <mailto:mbourvin at redhat.com
>>>         <mailto:mbourvin at redhat.com>>>>> wrote:
>>>
>>>                                                   Which error are you
>>>         getting
>>>                         in the UI?
>>>                                                   Can you please
>>>         attach the
>>>                         full engine and vdsm
>>>                                             logs?
>>>                                                   Also, please check
>>>         if there
>>>                         is a relevant
>>>                                             error in /var/log/messages
>>>
>>>
>>>          ------------------------------__----------------------------
>>> --__------------
>>>
>>>
>>>                                                       *From: *"Edgars M."
>>>                                             <edgars.mazurs at gmail.com
>>>         <mailto:edgars.mazurs at gmail.com>
>>>                         <mailto:edgars.mazurs at gmail.com
>>>         <mailto:edgars.mazurs at gmail.com>>
>>>                                                    <mailto:
>>> edgars.mazurs at gmail.com <mailto:edgars.mazurs at gmail.com>
>>>                         <mailto:edgars.mazurs at gmail.com
>>>         <mailto:edgars.mazurs at gmail.com>>>
>>>            <mailto:edgars.mazurs at gmail <mailto:edgars.mazurs at gmail>.
>>>                         <mailto:edgars.mazurs at gmail
>>>         <mailto:edgars.mazurs at gmail>.>__com
>>>
>>>                                                    <mailto:
>>> edgars.mazurs at gmail.com <mailto:edgars.mazurs at gmail.com>
>>>
>>>                         <mailto:edgars.mazurs at gmail.com
>>>         <mailto:edgars.mazurs at gmail.com>>>>>
>>>                                                       *To:
>>>         *users at ovirt.org <mailto:users at ovirt.org>
>>>                         <mailto:users at ovirt.org <mailto:users at ovirt.org
>>> >>
>>>                                             <mailto:users at ovirt.org
>>>         <mailto:users at ovirt.org>
>>>                         <mailto:users at ovirt.org
>>>         <mailto:users at ovirt.org>>> <mailto:users at ovirt.org
>>>
>>>         <mailto:users at ovirt.org>
>>>
>>>                         <mailto:users at ovirt.org <mailto:users at ovirt.org
>>> >>
>>>                                             <mailto:users at ovirt.org
>>>         <mailto:users at ovirt.org>
>>>                         <mailto:users at ovirt.org
>>>         <mailto:users at ovirt.org>>>>
>>>                                                       *Sent: *Friday,
>>>         January
>>>                         17, 2014 3:42:37 PM
>>>         *Subject: *[Users] VM
>>>                         Migration failed
>>>
>>>
>>>                                                       Hi
>>>
>>>                                                       I am
>>>         experiencing issues
>>>                         with manual VM
>>>                                             migration. VM fails to
>>>                                                       migrate to other
>>>         node in
>>>                         the same Cluster.
>>>                                             Here are some
>>>                                                       relevant
>>>         engine.log entries:
>>>
>>>                                                       ERROR
>>>          [org.ovirt.engine.core.__vdsbroker.__VdsUpdateRunTimeInfo]
>>>                         (DefaultQuartzScheduler___Worker-73) Rerun vm
>>>                         a31cfd62-26fc-4396-8a83-__1aed68c7fd39.
>>>                                             Called from vds novmnode1
>>>
>>>                                                       ERROR
>>>          [org.ovirt.engine.core.__vdsbroker.vdsbroker.__
>>> MigrateStatusVDSCommand]
>>>         (pool-6-thread-49) Failed in
>>>                                             MigrateStatusVDS method
>>>
>>>                                                       ERROR
>>>          [org.ovirt.engine.core.__vdsbroker.vdsbroker.__
>>> MigrateStatusVDSCommand]
>>>         (pool-6-thread-49) Error
>>>                         code migrateErr
>>>                                             and error message
>>>         VDSGenericException:
>>>                         VDSErrorException:
>>>                                             Failed to
>>>         MigrateStatusVDS, error
>>>                         = Fatal error
>>>                                             during migration
>>>
>>>                                                       ERROR
>>>          [org.ovirt.engine.core.__vdsbroker.vdsbroker.__
>>> MigrateStatusVDSCommand]
>>>         (pool-6-thread-49) Command
>>>                                             MigrateStatusVDS execution
>>>         failed.
>>>         Exception:
>>>                         VDSErrorException:
>>>         VDSGenericException:
>>>         VDSErrorException: Failed to
>>>                                             MigrateStatusVDS, error =
>>>         Fatal
>>>                                                       error during
>>>         migration
>>>
>>>                                                       Both Engine and
>>>         Node are
>>>                         running CentOS
>>>                                             6.5 x64. oVirt Engine
>>>                                                       Version:
>>>         3.3.2-1.el6.
>>>                         VDSM version
>>>                                             4.13.2-1. I restarted engine
>>>                                                       and vdsm, that
>>>         did not help.
>>>
>>>                                                       Also in vdsm log
>>>         file I
>>>                         see the following
>>>                                             errors related to the
>>>                                                       same VM ID:
>>>
>>>                         Thread-27::ERROR::2014-01-17
>>>
>>>                         16:37:06,271::sampling::355::__vm.Vm::(collect)
>>>
>>>                                vmId=`a31cfd62-26fc-4396-8a83-
>>> __1aed68c7fd39`::Stats
>>>                         function
>>>                                                       failed:
>>>                         <AdvancedStatsFunction _highWrite
>>>                                             at 0x26efb58>
>>>
>>>                                                       Any hints?
>>>
>>>                                                       BR
>>>                                                       Edgars
>>>
>>>
>>>                         ______________________________
>>> ___________________
>>>                                                       Users mailing list
>>>         Users at ovirt.org <mailto:Users at ovirt.org>
>>>         <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>>>                         <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org> <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>>>
>>>                                             <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>
>>>                         <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>> <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>
>>>                         <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>>>>
>>>         http://lists.ovirt.org/__mailman/listinfo/users
>>>          <http://lists.ovirt.org/mailman/listinfo/users>
>>>
>>>
>>>
>>>
>>>
>>>          _________________________________________________
>>>                                             Users mailing list
>>>         Users at ovirt.org <mailto:Users at ovirt.org>
>>>         <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>>>                         <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org> <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>>>
>>>         http://lists.ovirt.org/__mailman/listinfo/users
>>>          <http://lists.ovirt.org/mailman/listinfo/users>
>>>
>>>
>>>
>>>
>>>
>>>                                        ______________________________
>>> ___________________
>>>                                 Users mailing list
>>>         Users at ovirt.org <mailto:Users at ovirt.org>
>>>         <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>>>                         <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org> <mailto:Users at ovirt.org
>>>         <mailto:Users at ovirt.org>>>
>>>         http://lists.ovirt.org/__mailman/listinfo/users
>>>                                        <http://lists.ovirt.org/
>>> mailman/listinfo/users>
>>>
>>>
>>>
>>>                             --
>>>                             Dafna Ron
>>>
>>>
>>>
>>>
>>>             --     Dafna Ron
>>>
>>>
>>>
>>>
>>>     --     Dafna Ron
>>>
>>>
>>>
>>
>> --
>> Dafna Ron
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140120/013452d8/attachment-0001.html>


More information about the Users mailing list