[ovirt-users] memory leak in 3.5.6 - not vdsm
Charles Kozler
charles at fixflyer.com
Mon Feb 1 17:49:52 UTC 2016
So I will still have the memory leak?
On Feb 1, 2016 12:39 PM, "Simone Tiraboschi" <stirabos at redhat.com> wrote:
>
>
> On Mon, Feb 1, 2016 at 6:33 PM, Charles Kozler <charles at fixflyer.com>
> wrote:
>
>> So what about the bug that I hit for vdsm as listed above by Nir? Will I
>> have that patch to avoid the memory leak or no? Upgrading an entire node to
>> centos 7 is not actually feasible and was previously outlined above that I
>> just needed to upgrade to ovirt 3.6 and no mention of OS change ...
>>
>
> You cannot install VDSM from 3.6 on el6:
>
> http://www.ovirt.org/OVirt_3.6_Release_Notes#RHEL_6.7_-_CentOS_6.7_and_similar
> and there is no plan for a 3.5.8.
>
>
>
>> On Feb 1, 2016 12:30 PM, "Simone Tiraboschi" <stirabos at redhat.com> wrote:
>>
>>>
>>>
>>> On Mon, Feb 1, 2016 at 5:40 PM, Charles Kozler <charles at fixflyer.com>
>>> wrote:
>>>
>>>> Sandro / Nir -
>>>>
>>>> I followed your steps plus
>>>>
>>>> http://www.ovirt.org/OVirt_3.6_Release_Notes#Fedora_.2F_CentOS_.2F_RHEL
>>>>
>>>> Engine upgraded fine but then when I got to upgrading a node I did:
>>>>
>>>> $ yum install
>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm
>>>> $ yum update -y
>>>>
>>>> And then rebooted the node. I noticed libvirt was updated by a .1
>>>> release number but vdsm (where the memory leak issue I thought was?) was
>>>> not upgraded. In fact, very little of ovirt packages on the node were
>>>> noticeably not updated
>>>>
>>>>
>>> We are not building vdsm for el6 in 3.6, you need also to upgrade to el7
>>> if you want that.
>>>
>>>
>>>> Updated node received the following updated packages during the install:
>>>>
>>>> http://pastebin.ca/3362714
>>>>
>>>> Note specifically the only packages updated via the ovirt3.6 repository
>>>> was ioprocess, otopi, ovirt-engine-sdk-python, ovirt-host-deploy,
>>>> ovirt-release36, and python-ioprocess. I had expected to see some packages
>>>> like vdsm and the likes updated - or was this not the case?
>>>>
>>>> Upgraded node:
>>>>
>>>> [compute[root at node02 yum.repos.d]$ rpm -qa | grep -i vdsm
>>>> vdsm-4.16.30-0.el6.x86_64
>>>> vdsm-python-zombiereaper-4.16.30-0.el6.noarch
>>>> vdsm-cli-4.16.30-0.el6.noarch
>>>> vdsm-yajsonrpc-4.16.30-0.el6.noarch
>>>> vdsm-jsonrpc-4.16.30-0.el6.noarch
>>>> vdsm-xmlrpc-4.16.30-0.el6.noarch
>>>> vdsm-python-4.16.30-0.el6.noarch
>>>>
>>>> Nonupgraded node
>>>>
>>>> [compute[root at node01 ~]$ rpm -qa | grep -i vdsm
>>>> vdsm-cli-4.16.30-0.el6.noarch
>>>> vdsm-jsonrpc-4.16.30-0.el6.noarch
>>>> vdsm-python-zombiereaper-4.16.30-0.el6.noarch
>>>> vdsm-xmlrpc-4.16.30-0.el6.noarch
>>>> vdsm-yajsonrpc-4.16.30-0.el6.noarch
>>>> vdsm-4.16.30-0.el6.x86_64
>>>> vdsm-python-4.16.30-0.el6.noarch
>>>>
>>>> Also, the docs stated that the engine VM would migrate to the freshly
>>>> upgraded node since it would have a higher number but it did not
>>>>
>>>> So I cant really confirm whether or not my issue will be resolved? Or
>>>> that if the node was actually updated properly?
>>>>
>>>> Please advise on how to confirm
>>>>
>>>> Thank you!
>>>>
>>>> On Sat, Jan 23, 2016 at 12:55 AM, Charles Kozler <charles at fixflyer.com>
>>>> wrote:
>>>>
>>>>> Thanks Sandro. Should clarify my storage is external on a redundant
>>>>> SAN. The steps I was concerned about was the actual upgrade. I tried to
>>>>> upgrade before and it brought my entire stack crumbling down so I'm
>>>>> hesitant. This bug seems like a huge bug that should at least somehow
>>>>> backported if at all possible because, to me, it renders the entire 3.5.6
>>>>> branch unusable as no VMs can be deployed since OOM will eventually kill
>>>>> them. In any case that's just my opinion and I'm a new user to ovirt. The
>>>>> docs I followed originally got me going how I need and somehow didn't work
>>>>> for 3.6 in the same fashion so naturally I'm hesitant to upgrade but
>>>>> clearly have no option if I want to continue my infrastructure on ovirt.
>>>>> Thank you again for taking the time out to assist me, I truly appreciate
>>>>> it. I will try an upgrade next week and pray it all goes well :-)
>>>>> On Jan 23, 2016 12:40 AM, "Sandro Bonazzola" <sbonazzo at redhat.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 10:53 PM, Charles Kozler <
>>>>>> charles at fixflyer.com> wrote:
>>>>>>
>>>>>>> Sandro -
>>>>>>>
>>>>>>> Do you have available documentation that can support upgrading self
>>>>>>> hosted? I followed this
>>>>>>> http://community.redhat.com/blog/2014/10/up-and-running-with-ovirt-3-5/
>>>>>>>
>>>>>>> Would it be as easy as installing the RPM and then running yum
>>>>>>> upgrade?
>>>>>>>
>>>>>>>
>>>>>> Note that mentioned article describes an unsupported hyperconverged
>>>>>> setup running NFS over Gluster.
>>>>>> That said,
>>>>>> 1) put the hosted-engine storage domain into global maintenance mode
>>>>>> 2) upgrade the engine VM
>>>>>> 3) select the first host to upgrade and put it under maintenance from
>>>>>> the engine, wait for the engine vm to migrate if needed.
>>>>>> 4) yum upgrade the first host and wait until ovirt-ha-agent completes
>>>>>> 5) exit global and local maintenance mode
>>>>>> 6) repeat 3-5 on all the other hosts
>>>>>> 7) once all hosts are updated you can increase the cluster
>>>>>> compatibility level to 3.6. At this point the engine will trigger the
>>>>>> auto-import of the hosted-engine storage domain.
>>>>>>
>>>>>> Simone, Roy, can you confirm above steps? Maybe also you can update
>>>>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 4:42 PM, Sandro Bonazzola <
>>>>>>> sbonazzo at redhat.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Il 22/Gen/2016 22:31, "Charles Kozler" <charles at fixflyer.com> ha
>>>>>>>> scritto:
>>>>>>>> >
>>>>>>>> > Hi Nir -
>>>>>>>> >
>>>>>>>> > do you have a release target date for 3.5.8? Any estimate would
>>>>>>>> help.
>>>>>>>> >
>>>>>>>>
>>>>>>>> There won't be any supported release after 3.5.6. Please update to
>>>>>>>> 3.6.2 next week
>>>>>>>>
>>>>>>>> > If its not VDSM, what is it exactly? Sorry, I understood from the
>>>>>>>> ticket it was something inside vdsm, was I mistaken?
>>>>>>>> >
>>>>>>>> > CentOS 6 is the servers. 6.7 to be exact
>>>>>>>> >
>>>>>>>> > I have done all forms of flushing that I can (page cache, inodes,
>>>>>>>> dentry's, etc) and as well moved VM's around to other nodes and nothing
>>>>>>>> changes the memory. How can I find the leak? Where is the leak? RES shows
>>>>>>>> the following of which, the totals dont add up to 20GB
>>>>>>>> >
>>>>>>>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>>>>>>> COMMAND
>>>>>>>>
>>>>>>>> > 19044 qemu 20 0 8876m 4.0g 5680 S 3.6 12.9 1571:44
>>>>>>>> qemu-kvm
>>>>>>>>
>>>>>>>> > 26143 qemu 20 0 5094m 1.1g 5624 S 9.2 3.7 6012:12
>>>>>>>> qemu-kvm
>>>>>>>>
>>>>>>>> > 5837 root 0 -20 964m 624m 3664 S 0.0 2.0 85:22.09
>>>>>>>> glusterfs
>>>>>>>>
>>>>>>>> > 14328 root 0 -20 635m 169m 3384 S 0.0 0.5 43:15.23
>>>>>>>> glusterfs
>>>>>>>>
>>>>>>>> > 5134 vdsm 0 -20 4368m 111m 10m S 5.9 0.3 3710:50
>>>>>>>> vdsm
>>>>>>>>
>>>>>>>> > 4095 root 15 -5 727m 43m 10m S 0.0 0.1 0:02.00
>>>>>>>> supervdsmServer
>>>>>>>> >
>>>>>>>> > 4.0G + 1.1G + 624M + 169 + 111M + 43M = ~7GB
>>>>>>>> >
>>>>>>>> > This was top sorted by RES from highest to lowest
>>>>>>>> >
>>>>>>>> > At that point I wouldnt know where else to look except slab /
>>>>>>>> kernel structures. Of which slab shows:
>>>>>>>> >
>>>>>>>> > [compute[root at node1 ~]$ cat /proc/meminfo | grep -i slab
>>>>>>>> > Slab: 2549748 kB
>>>>>>>> >
>>>>>>>> > So roughly 2-3GB. Adding that to the other use of 7GB we have
>>>>>>>> still about 10GB unaccounted for
>>>>>>>> >
>>>>>>>> > On Fri, Jan 22, 2016 at 4:24 PM, Nir Soffer <nsoffer at redhat.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler <
>>>>>>>> charles at fixflyer.com> wrote:
>>>>>>>> >> > Hi Nir -
>>>>>>>> >> >
>>>>>>>> >> > Thanks for getting back to me. Will the patch to 3.6 be
>>>>>>>> backported to 3.5?
>>>>>>>> >>
>>>>>>>> >> We plan to include them in 3.5.8.
>>>>>>>> >>
>>>>>>>> >> > As you can tell from the images, it takes days and days for it
>>>>>>>> to increase
>>>>>>>> >> > over time. I also wasnt sure if that was the right bug because
>>>>>>>> VDSM memory
>>>>>>>> >> > shows normal from top ...
>>>>>>>> >> >
>>>>>>>> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>>>>>>> COMMAND
>>>>>>>> >> > 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28
>>>>>>>> vdsm
>>>>>>>> >>
>>>>>>>> >> As you wrote, this issue is not related to vdsm.
>>>>>>>> >>
>>>>>>>> >> >
>>>>>>>> >> > Res is only 111M. This is from node1 which is showing
>>>>>>>> currently 20GB of 32GB
>>>>>>>> >> > used with only 2 VMs running on it - 1 with 4G and another
>>>>>>>> with ~1 GB of RAM
>>>>>>>> >> > configured
>>>>>>>> >> >
>>>>>>>> >> > The images are from nagios and the value here is a direct
>>>>>>>> correlation to
>>>>>>>> >> > what you would see in the free command output. See below from
>>>>>>>> an example of
>>>>>>>> >> > node 1 and node 2
>>>>>>>> >> >
>>>>>>>> >> > [compute[root at node1 ~]$ free
>>>>>>>> >> > total used free shared
>>>>>>>> buffers cached
>>>>>>>> >> > Mem: 32765316 20318156 12447160 252
>>>>>>>> 30884 628948
>>>>>>>> >> > -/+ buffers/cache: 19658324 13106992
>>>>>>>> >> > Swap: 19247100 0 19247100
>>>>>>>> >> > [compute[root at node1 ~]$ free -m
>>>>>>>> >> > total used free shared
>>>>>>>> buffers cached
>>>>>>>> >> > Mem: 31997 19843 12153 0
>>>>>>>> 30 614
>>>>>>>> >> > -/+ buffers/cache: 19199 12798
>>>>>>>> >> > Swap: 18795 0 18795
>>>>>>>> >> >
>>>>>>>> >> > And its correlated image http://i.imgur.com/PZLEgyx.png
>>>>>>>> (~19GB used)
>>>>>>>> >> >
>>>>>>>> >> > And as a control, node 2 that I just restarted today
>>>>>>>> >> >
>>>>>>>> >> > [compute[root at node2 ~]$ free
>>>>>>>> >> > total used free shared
>>>>>>>> buffers cached
>>>>>>>> >> > Mem: 32765316 1815324 30949992 212
>>>>>>>> 35784 717320
>>>>>>>> >> > -/+ buffers/cache: 1062220 31703096
>>>>>>>> >> > Swap: 19247100 0 19247100
>>>>>>>> >>
>>>>>>>> >> Is this rhel/centos 6?
>>>>>>>> >>
>>>>>>>> >> > [compute[root at node2 ~]$ free -m
>>>>>>>> >> > total used free shared
>>>>>>>> buffers cached
>>>>>>>> >> > Mem: 31997 1772 30225 0
>>>>>>>> 34 700
>>>>>>>> >> > -/+ buffers/cache: 1036 30960
>>>>>>>> >> > Swap: 18795 0 18795
>>>>>>>> >> >
>>>>>>>> >> > And its correlated image http://i.imgur.com/8ldPVqY.png
>>>>>>>> (~2GB used). Note
>>>>>>>> >> > how 1772 in the image is exactly what is registered under
>>>>>>>> 'used' in free
>>>>>>>> >> > command
>>>>>>>> >>
>>>>>>>> >> I guess you should start looking at the processes running on
>>>>>>>> these nodes.
>>>>>>>> >>
>>>>>>>> >> Maybe try to collect memory usage per process using ps?
>>>>>>>> >>
>>>>>>>> >> >
>>>>>>>> >> > On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer <
>>>>>>>> nsoffer at redhat.com> wrote:
>>>>>>>> >> >>
>>>>>>>> >> >> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler <
>>>>>>>> charles at fixflyer.com>
>>>>>>>> >> >> wrote:
>>>>>>>> >> >> > Here is a screenshot of my three nodes and their increased
>>>>>>>> memory usage
>>>>>>>> >> >> > over
>>>>>>>> >> >> > 30 days. Note that node #2 had 1 single VM that had 4GB of
>>>>>>>> RAM assigned
>>>>>>>> >> >> > to
>>>>>>>> >> >> > it. I had since shut it down and saw no memory reclamation
>>>>>>>> occur.
>>>>>>>> >> >> > Further, I
>>>>>>>> >> >> > flushed page caches and inodes and ran 'sync'. I tried
>>>>>>>> everything but
>>>>>>>> >> >> > nothing brought the memory usage down. vdsm was low too
>>>>>>>> (couple hundred
>>>>>>>> >> >> > MB)
>>>>>>>> >> >>
>>>>>>>> >> >> Note that there is an old leak in vdsm, will be fixed in next
>>>>>>>> 3.6 build:
>>>>>>>> >> >> https://bugzilla.redhat.com/1269424
>>>>>>>> >> >>
>>>>>>>> >> >> > and there was no qemu-kvm process running so I'm at a loss
>>>>>>>> >> >> >
>>>>>>>> >> >> > http://imgur.com/a/aFPcK
>>>>>>>> >> >> >
>>>>>>>> >> >> > Please advise on what I can do to debug this. Note I have
>>>>>>>> restarted node
>>>>>>>> >> >> > 2
>>>>>>>> >> >> > (which is why you see the drop) to see if it raises in
>>>>>>>> memory use over
>>>>>>>> >> >> > tim
>>>>>>>> >> >> > even with no VM's running
>>>>>>>> >> >>
>>>>>>>> >> >> Not sure what is "memory" that you show in the graphs.
>>>>>>>> Theoretically this
>>>>>>>> >> >> may be
>>>>>>>> >> >> normal memory usage, Linux using free memory for the buffer
>>>>>>>> cache.
>>>>>>>> >> >>
>>>>>>>> >> >> Can you instead show the output of "free", during one day,
>>>>>>>> maybe run once
>>>>>>>> >> >> per hour?
>>>>>>>> >> >>
>>>>>>>> >> >> You may also like to install sysstat for collecting and
>>>>>>>> monitoring
>>>>>>>> >> >> resources usage.
>>>>>>>> >> >>
>>>>>>>> >> >> >
>>>>>>>> >> >> > [compute[root at node2 log]$ rpm -qa | grep -i ovirt
>>>>>>>> >> >> > libgovirt-0.3.2-1.el6.x86_64
>>>>>>>> >> >> > ovirt-release35-006-1.noarch
>>>>>>>> >> >> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch
>>>>>>>> >> >> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch
>>>>>>>> >> >> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch
>>>>>>>> >> >> > ovirt-host-deploy-1.3.2-1.el6.noarch
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > --
>>>>>>>> >> >> >
>>>>>>>> >> >> > Charles Kozler
>>>>>>>> >> >> > Vice President, IT Operations
>>>>>>>> >> >> >
>>>>>>>> >> >> > FIX Flyer, LLC
>>>>>>>> >> >> > 225 Broadway | Suite 1600 | New York, NY 10007
>>>>>>>> >> >> > 1-888-349-3593
>>>>>>>> >> >> > http://www.fixflyer.com
>>>>>>>> >> >> >
>>>>>>>> >> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE
>>>>>>>> INTENDED
>>>>>>>> >> >> > RECIPIENT(S)
>>>>>>>> >> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>>>>>>>> WHICH IS
>>>>>>>> >> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE,
>>>>>>>> COPYING,
>>>>>>>> >> >> > DISTRIBUTION,
>>>>>>>> >> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
>>>>>>>> THIS INFORMATION
>>>>>>>> >> >> > IS
>>>>>>>> >> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>>>>>>>> RECIPIENT,
>>>>>>>> >> >> > PLEASE
>>>>>>>> >> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS
>>>>>>>> E-MAIL FROM
>>>>>>>> >> >> > YOUR
>>>>>>>> >> >> > SYSTEM AND DESTROY ANY COPIES.
>>>>>>>> >> >> >
>>>>>>>> >> >> > _______________________________________________
>>>>>>>> >> >> > Users mailing list
>>>>>>>> >> >> > Users at ovirt.org
>>>>>>>> >> >> > http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>> >> >> >
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> > --
>>>>>>>> >> >
>>>>>>>> >> > Charles Kozler
>>>>>>>> >> > Vice President, IT Operations
>>>>>>>> >> >
>>>>>>>> >> > FIX Flyer, LLC
>>>>>>>> >> > 225 Broadway | Suite 1600 | New York, NY 10007
>>>>>>>> >> > 1-888-349-3593
>>>>>>>> >> > http://www.fixflyer.com
>>>>>>>> >> >
>>>>>>>> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE
>>>>>>>> INTENDED RECIPIENT(S)
>>>>>>>> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>>>>>>>> WHICH IS
>>>>>>>> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING,
>>>>>>>> DISTRIBUTION,
>>>>>>>> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
>>>>>>>> INFORMATION IS
>>>>>>>> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>>>>>>>> RECIPIENT, PLEASE
>>>>>>>> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS
>>>>>>>> E-MAIL FROM YOUR
>>>>>>>> >> > SYSTEM AND DESTROY ANY COPIES.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> >
>>>>>>>> > Charles Kozler
>>>>>>>> > Vice President, IT Operations
>>>>>>>> >
>>>>>>>> > FIX Flyer, LLC
>>>>>>>> > 225 Broadway | Suite 1600 | New York, NY 10007
>>>>>>>> > 1-888-349-3593
>>>>>>>> > http://www.fixflyer.com
>>>>>>>> >
>>>>>>>> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED
>>>>>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>>>>>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING,
>>>>>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
>>>>>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>>>>>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS
>>>>>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > Users mailing list
>>>>>>>> > Users at ovirt.org
>>>>>>>> > http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *Charles Kozler*
>>>>>>> *Vice President, IT Operations*
>>>>>>>
>>>>>>> FIX Flyer, LLC
>>>>>>> 225 Broadway | Suite 1600 | New York, NY 10007
>>>>>>> 1-888-349-3593
>>>>>>> http://www.fixflyer.com <http://fixflyer.com>
>>>>>>>
>>>>>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED
>>>>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>>>>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING,
>>>>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
>>>>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>>>>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS
>>>>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sandro Bonazzola
>>>>>> Better technology. Faster innovation. Powered by community
>>>>>> collaboration.
>>>>>> See how it works at redhat.com
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Charles Kozler*
>>>> *Vice President, IT Operations*
>>>>
>>>> FIX Flyer, LLC
>>>> 225 Broadway | Suite 1600 | New York, NY 10007
>>>> 1-888-349-3593
>>>> http://www.fixflyer.com <http://fixflyer.com>
>>>>
>>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED
>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING,
>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS
>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160201/1c396e2c/attachment-0001.html>
More information about the Users
mailing list