Re: [ovirt-users] memory leak in 3.5.6 - not vdsm

Sorry to be clear..the only way to resolve the memory leak I am facing now is to upgrade to el7?

Also the engine can stay running on el6 yes?

I successfully upgraded my engine to ovirt 3.6 in el6. Do I need to make my engine vm el7 too?

On Feb 1, 2016 12:49 PM, "Charles Kozler" <charles(a)fixflyer.com&gt; wrote: > > So I will still have the memory leak? > > On Feb 1, 2016 12:39 PM, "Simone Tiraboschi" <stirabos(a)redhat.com&gt; wrote: >> >> >> >> On Mon, Feb 1, 2016 at 6:33 PM, Charles Kozler <charles(a)fixflyer.com&gt; >> wrote: >>> >>> So what about the bug that I hit for vdsm as listed above by Nir? Will I >>> have that patch to avoid the memory leak or no? Upgrading an entire node to >>> centos 7 is not actually feasible and was previously outlined above that I >>> just needed to upgrade to ovirt 3.6 and no mention of OS change ... >> >> >> You cannot install VDSM from 3.6 on el6: >> >> http://www.ovirt.org/OVirt_3.6_Release_Notes#RHEL_6.7_-_CentOS_6.7_and_si... >> and there is no plan for a 3.5.8. >> >> >>> >>> On Feb 1, 2016 12:30 PM, "Simone Tiraboschi" <stirabos(a)redhat.com&gt; >>> wrote: >>>> >>>> >>>> >>>> On Mon, Feb 1, 2016 at 5:40 PM, Charles Kozler <charles(a)fixflyer.com&gt; >>>> wrote: >>>>> >>>>> Sandro / Nir - >>>>> >>>>> I followed your steps plus >>>>> >>>>> >>>>> http://www.ovirt.org/OVirt_3.6_Release_Notes#Fedora_.2F_CentOS_.2F_RHEL >>>>> >>>>> Engine upgraded fine but then when I got to upgrading a node I did: >>>>> >>>>> $ yum install >>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm >>>>> $ yum update -y >>>>> >>>>> And then rebooted the node. I noticed libvirt was updated by a .1 >>>>> release number but vdsm (where the memory leak issue I thought was?) was not >>>>> upgraded. In fact, very little of ovirt packages on the node were noticeably >>>>> not updated >>>>> >>>> >>>> We are not building vdsm for el6 in 3.6, you need also to upgrade to >>>> el7 if you want that. >>>> >>>>> >>>>> Updated node received the following updated packages during the >>>>> install: >>>>> >>>>> http://pastebin.ca/3362714 >>>>> >>>>> Note specifically the only packages updated via the ovirt3.6 >>>>> repository was ioprocess, otopi, ovirt-engine-sdk-python, ovirt-host-deploy, >>>>> ovirt-release36, and python-ioprocess. I had expected to see some packages >>>>> like vdsm and the likes updated - or was this not the case? >>>>> >>>>> Upgraded node: >>>>> >>>>> [compute[root@node02 yum.repos.d]$ rpm -qa | grep -i vdsm >>>>> vdsm-4.16.30-0.el6.x86_64 >>>>> vdsm-python-zombiereaper-4.16.30-0.el6.noarch >>>>> vdsm-cli-4.16.30-0.el6.noarch >>>>> vdsm-yajsonrpc-4.16.30-0.el6.noarch >>>>> vdsm-jsonrpc-4.16.30-0.el6.noarch >>>>> vdsm-xmlrpc-4.16.30-0.el6.noarch >>>>> vdsm-python-4.16.30-0.el6.noarch >>>>> >>>>> Nonupgraded node >>>>> >>>>> [compute[root@node01 ~]$ rpm -qa | grep -i vdsm >>>>> vdsm-cli-4.16.30-0.el6.noarch >>>>> vdsm-jsonrpc-4.16.30-0.el6.noarch >>>>> vdsm-python-zombiereaper-4.16.30-0.el6.noarch >>>>> vdsm-xmlrpc-4.16.30-0.el6.noarch >>>>> vdsm-yajsonrpc-4.16.30-0.el6.noarch >>>>> vdsm-4.16.30-0.el6.x86_64 >>>>> vdsm-python-4.16.30-0.el6.noarch >>>>> >>>>> Also, the docs stated that the engine VM would migrate to the freshly >>>>> upgraded node since it would have a higher number but it did not >>>>> >>>>> So I cant really confirm whether or not my issue will be resolved? Or >>>>> that if the node was actually updated properly? >>>>> >>>>> Please advise on how to confirm >>>>> >>>>> Thank you! >>>>> >>>>> On Sat, Jan 23, 2016 at 12:55 AM, Charles Kozler >>>>> <charles(a)fixflyer.com&gt; wrote: >>>>>> >>>>>> Thanks Sandro. Should clarify my storage is external on a redundant >>>>>> SAN. The steps I was concerned about was the actual upgrade. I tried to >>>>>> upgrade before and it brought my entire stack crumbling down so I'm >>>>>> hesitant. This bug seems like a huge bug that should at least somehow >>>>>> backported if at all possible because, to me, it renders the entire 3.5.6 >>>>>> branch unusable as no VMs can be deployed since OOM will eventually kill >>>>>> them. In any case that's just my opinion and I'm a new user to ovirt. The >>>>>> docs I followed originally got me going how I need and somehow didn't work >>>>>> for 3.6 in the same fashion so naturally I'm hesitant to upgrade but clearly >>>>>> have no option if I want to continue my infrastructure on ovirt. Thank you >>>>>> again for taking the time out to assist me, I truly appreciate it. I will >>>>>> try an upgrade next week and pray it all goes well :-) >>>>>> >>>>>> On Jan 23, 2016 12:40 AM, "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 22, 2016 at 10:53 PM, Charles Kozler >>>>>>> <charles(a)fixflyer.com&gt; wrote: >>>>>>>> >>>>>>>> Sandro - >>>>>>>> >>>>>>>> Do you have available documentation that can support upgrading self >>>>>>>> hosted? I followed this >>>>>>>> http://community.redhat.com/blog/2014/10/up-and-running-with-ovirt-3-5/ >>>>>>>> >>>>>>>> Would it be as easy as installing the RPM and then running yum >>>>>>>> upgrade? >>>>>>>> >>>>>>> >>>>>>> Note that mentioned article describes an unsupported hyperconverged >>>>>>> setup running NFS over Gluster. >>>>>>> That said, >>>>>>> 1) put the hosted-engine storage domain into global maintenance mode >>>>>>> 2) upgrade the engine VM >>>>>>> 3) select the first host to upgrade and put it under maintenance >>>>>>> from the engine, wait for the engine vm to migrate if needed. >>>>>>> 4) yum upgrade the first host and wait until ovirt-ha-agent >>>>>>> completes >>>>>>> 5) exit global and local maintenance mode >>>>>>> 6) repeat 3-5 on all the other hosts >>>>>>> 7) once all hosts are updated you can increase the cluster >>>>>>> compatibility level to 3.6. At this point the engine will trigger the >>>>>>> auto-import of the hosted-engine storage domain. >>>>>>> >>>>>>> Simone, Roy, can you confirm above steps? Maybe also you can update >>>>>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> On Fri, Jan 22, 2016 at 4:42 PM, Sandro Bonazzola >>>>>>>> <sbonazzo(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Il 22/Gen/2016 22:31, "Charles Kozler" <charles(a)fixflyer.com&gt; ha >>>>>>>>> scritto: >>>>>>>>> > >>>>>>>>> > Hi Nir - >>>>>>>>> > >>>>>>>>> > do you have a release target date for 3.5.8? Any estimate would >>>>>>>>> > help. >>>>>>>>> > >>>>>>>>> >>>>>>>>> There won't be any supported release after 3.5.6. Please update to >>>>>>>>> 3.6.2 next week >>>>>>>>> >>>>>>>>> > If its not VDSM, what is it exactly? Sorry, I understood from >>>>>>>>> > the ticket it was something inside vdsm, was I mistaken? >>>>>>>>> > >>>>>>>>> > CentOS 6 is the servers. 6.7 to be exact >>>>>>>>> > >>>>>>>>> > I have done all forms of flushing that I can (page cache, >>>>>>>>> > inodes, dentry's, etc) and as well moved VM's around to other nodes and >>>>>>>>> > nothing changes the memory. How can I find the leak? Where is the leak? RES >>>>>>>>> > shows the following of which, the totals dont add up to 20GB >>>>>>>>> > >>>>>>>>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>>>> > COMMAND >>>>>>>>> > 19044 qemu 20 0 8876m 4.0g 5680 S 3.6 12.9 1571:44 >>>>>>>>> > qemu-kvm >>>>>>>>> > 26143 qemu 20 0 5094m 1.1g 5624 S 9.2 3.7 6012:12 >>>>>>>>> > qemu-kvm >>>>>>>>> > 5837 root 0 -20 964m 624m 3664 S 0.0 2.0 85:22.09 >>>>>>>>> > glusterfs >>>>>>>>> > 14328 root 0 -20 635m 169m 3384 S 0.0 0.5 43:15.23 >>>>>>>>> > glusterfs >>>>>>>>> > 5134 vdsm 0 -20 4368m 111m 10m S 5.9 0.3 3710:50 >>>>>>>>> > vdsm >>>>>>>>> > 4095 root 15 -5 727m 43m 10m S 0.0 0.1 0:02.00 >>>>>>>>> > supervdsmServer >>>>>>>>> > >>>>>>>>> > 4.0G + 1.1G + 624M + 169 + 111M + 43M = ~7GB >>>>>>>>> > >>>>>>>>> > This was top sorted by RES from highest to lowest >>>>>>>>> > >>>>>>>>> > At that point I wouldnt know where else to look except slab / >>>>>>>>> > kernel structures. Of which slab shows: >>>>>>>>> > >>>>>>>>> > [compute[root@node1 ~]$ cat /proc/meminfo | grep -i slab >>>>>>>>> > Slab: 2549748 kB >>>>>>>>> > >>>>>>>>> > So roughly 2-3GB. Adding that to the other use of 7GB we have >>>>>>>>> > still about 10GB unaccounted for >>>>>>>>> > >>>>>>>>> > On Fri, Jan 22, 2016 at 4:24 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>>>>> > wrote: >>>>>>>>> >> >>>>>>>>> >> On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler >>>>>>>>> >> <charles(a)fixflyer.com&gt; wrote: >>>>>>>>> >> > Hi Nir - >>>>>>>>> >> > >>>>>>>>> >> > Thanks for getting back to me. Will the patch to 3.6 be >>>>>>>>> >> > backported to 3.5? >>>>>>>>> >> >>>>>>>>> >> We plan to include them in 3.5.8. >>>>>>>>> >> >>>>>>>>> >> > As you can tell from the images, it takes days and days for >>>>>>>>> >> > it to increase >>>>>>>>> >> > over time. I also wasnt sure if that was the right bug >>>>>>>>> >> > because VDSM memory >>>>>>>>> >> > shows normal from top ... >>>>>>>>> >> > >>>>>>>>> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>>>> >> > COMMAND >>>>>>>>> >> > 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28 >>>>>>>>> >> > vdsm >>>>>>>>> >> >>>>>>>>> >> As you wrote, this issue is not related to vdsm. >>>>>>>>> >> >>>>>>>>> >> > >>>>>>>>> >> > Res is only 111M. This is from node1 which is showing >>>>>>>>> >> > currently 20GB of 32GB >>>>>>>>> >> > used with only 2 VMs running on it - 1 with 4G and another >>>>>>>>> >> > with ~1 GB of RAM >>>>>>>>> >> > configured >>>>>>>>> >> > >>>>>>>>> >> > The images are from nagios and the value here is a direct >>>>>>>>> >> > correlation to >>>>>>>>> >> > what you would see in the free command output. See below from >>>>>>>>> >> > an example of >>>>>>>>> >> > node 1 and node 2 >>>>>>>>> >> > >>>>>>>>> >> > [compute[root@node1 ~]$ free >>>>>>>>> >> > total used free shared >>>>>>>>> >> > buffers cached >>>>>>>>> >> > Mem: 32765316 20318156 12447160 252 >>>>>>>>> >> > 30884 628948 >>>>>>>>> >> > -/+ buffers/cache: 19658324 13106992 >>>>>>>>> >> > Swap: 19247100 0 19247100 >>>>>>>>> >> > [compute[root@node1 ~]$ free -m >>>>>>>>> >> > total used free shared >>>>>>>>> >> > buffers cached >>>>>>>>> >> > Mem: 31997 19843 12153 0 >>>>>>>>> >> > 30 614 >>>>>>>>> >> > -/+ buffers/cache: 19199 12798 >>>>>>>>> >> > Swap: 18795 0 18795 >>>>>>>>> >> > >>>>>>>>> >> > And its correlated image http://i.imgur.com/PZLEgyx.png >>>>>>>>> >> > (~19GB used) >>>>>>>>> >> > >>>>>>>>> >> > And as a control, node 2 that I just restarted today >>>>>>>>> >> > >>>>>>>>> >> > [compute[root@node2 ~]$ free >>>>>>>>> >> > total used free shared >>>>>>>>> >> > buffers cached >>>>>>>>> >> > Mem: 32765316 1815324 30949992 212 >>>>>>>>> >> > 35784 717320 >>>>>>>>> >> > -/+ buffers/cache: 1062220 31703096 >>>>>>>>> >> > Swap: 19247100 0 19247100 >>>>>>>>> >> >>>>>>>>> >> Is this rhel/centos 6? >>>>>>>>> >> >>>>>>>>> >> > [compute[root@node2 ~]$ free -m >>>>>>>>> >> > total used free shared >>>>>>>>> >> > buffers cached >>>>>>>>> >> > Mem: 31997 1772 30225 0 >>>>>>>>> >> > 34 700 >>>>>>>>> >> > -/+ buffers/cache: 1036 30960 >>>>>>>>> >> > Swap: 18795 0 18795 >>>>>>>>> >> > >>>>>>>>> >> > And its correlated image http://i.imgur.com/8ldPVqY.png >>>>>>>>> >> > (~2GB used). Note >>>>>>>>> >> > how 1772 in the image is exactly what is registered under >>>>>>>>> >> > 'used' in free >>>>>>>>> >> > command >>>>>>>>> >> >>>>>>>>> >> I guess you should start looking at the processes running on >>>>>>>>> >> these nodes. >>>>>>>>> >> >>>>>>>>> >> Maybe try to collect memory usage per process using ps? >>>>>>>>> >> >>>>>>>>> >> > >>>>>>>>> >> > On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer >>>>>>>>> >> > <nsoffer(a)redhat.com&gt; wrote: >>>>>>>>> >> >> >>>>>>>>> >> >> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler >>>>>>>>> >> >> <charles(a)fixflyer.com&gt; >>>>>>>>> >> >> wrote: >>>>>>>>> >> >> > Here is a screenshot of my three nodes and their increased >>>>>>>>> >> >> > memory usage >>>>>>>>> >> >> > over >>>>>>>>> >> >> > 30 days. Note that node #2 had 1 single VM that had 4GB of >>>>>>>>> >> >> > RAM assigned >>>>>>>>> >> >> > to >>>>>>>>> >> >> > it. I had since shut it down and saw no memory reclamation >>>>>>>>> >> >> > occur. >>>>>>>>> >> >> > Further, I >>>>>>>>> >> >> > flushed page caches and inodes and ran 'sync'. I tried >>>>>>>>> >> >> > everything but >>>>>>>>> >> >> > nothing brought the memory usage down. vdsm was low too >>>>>>>>> >> >> > (couple hundred >>>>>>>>> >> >> > MB) >>>>>>>>> >> >> >>>>>>>>> >> >> Note that there is an old leak in vdsm, will be fixed in >>>>>>>>> >> >> next 3.6 build: >>>>>>>>> >> >> https://bugzilla.redhat.com/1269424 >>>>>>>>> >> >> >>>>>>>>> >> >> > and there was no qemu-kvm process running so I'm at a loss >>>>>>>>> >> >> > >>>>>>>>> >> >> > http://imgur.com/a/aFPcK >>>>>>>>> >> >> > >>>>>>>>> >> >> > Please advise on what I can do to debug this. Note I have >>>>>>>>> >> >> > restarted node >>>>>>>>> >> >> > 2 >>>>>>>>> >> >> > (which is why you see the drop) to see if it raises in >>>>>>>>> >> >> > memory use over >>>>>>>>> >> >> > tim >>>>>>>>> >> >> > even with no VM's running >>>>>>>>> >> >> >>>>>>>>> >> >> Not sure what is "memory" that you show in the graphs. >>>>>>>>> >> >> Theoretically this >>>>>>>>> >> >> may be >>>>>>>>> >> >> normal memory usage, Linux using free memory for the buffer >>>>>>>>> >> >> cache. >>>>>>>>> >> >> >>>>>>>>> >> >> Can you instead show the output of "free", during one day, >>>>>>>>> >> >> maybe run once >>>>>>>>> >> >> per hour? >>>>>>>>> >> >> >>>>>>>>> >> >> You may also like to install sysstat for collecting and >>>>>>>>> >> >> monitoring >>>>>>>>> >> >> resources usage. >>>>>>>>> >> >> >>>>>>>>> >> >> > >>>>>>>>> >> >> > [compute[root@node2 log]$ rpm -qa | grep -i ovirt >>>>>>>>> >> >> > libgovirt-0.3.2-1.el6.x86_64 >>>>>>>>> >> >> > ovirt-release35-006-1.noarch >>>>>>>>> >> >> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch >>>>>>>>> >> >> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch >>>>>>>>> >> >> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch >>>>>>>>> >> >> > ovirt-host-deploy-1.3.2-1.el6.noarch >>>>>>>>> >> >> > >>>>>>>>> >> >> > >>>>>>>>> >> >> > -- >>>>>>>>> >> >> > >>>>>>>>> >> >> > Charles Kozler >>>>>>>>> >> >> > Vice President, IT Operations >>>>>>>>> >> >> > >>>>>>>>> >> >> > FIX Flyer, LLC >>>>>>>>> >> >> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>>> >> >> > 1-888-349-3593 >>>>>>>>> >> >> > http://www.fixflyer.com >>>>>>>>> >> >> > >>>>>>>>> >> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE >>>>>>>>> >> >> > INTENDED >>>>>>>>> >> >> > RECIPIENT(S) >>>>>>>>> >> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>>> >> >> > WHICH IS >>>>>>>>> >> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, >>>>>>>>> >> >> > COPYING, >>>>>>>>> >> >> > DISTRIBUTION, >>>>>>>>> >> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO >>>>>>>>> >> >> > THIS INFORMATION >>>>>>>>> >> >> > IS >>>>>>>>> >> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>>> >> >> > RECIPIENT, >>>>>>>>> >> >> > PLEASE >>>>>>>>> >> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>>> >> >> > E-MAIL FROM >>>>>>>>> >> >> > YOUR >>>>>>>>> >> >> > SYSTEM AND DESTROY ANY COPIES. >>>>>>>>> >> >> > >>>>>>>>> >> >> > _______________________________________________ >>>>>>>>> >> >> > Users mailing list >>>>>>>>> >> >> > Users(a)ovirt.org >>>>>>>>> >> >> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> >> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > -- >>>>>>>>> >> > >>>>>>>>> >> > Charles Kozler >>>>>>>>> >> > Vice President, IT Operations >>>>>>>>> >> > >>>>>>>>> >> > FIX Flyer, LLC >>>>>>>>> >> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>>> >> > 1-888-349-3593 >>>>>>>>> >> > http://www.fixflyer.com >>>>>>>>> >> > >>>>>>>>> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE >>>>>>>>> >> > INTENDED RECIPIENT(S) >>>>>>>>> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>>> >> > WHICH IS >>>>>>>>> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>>>>> >> > DISTRIBUTION, >>>>>>>>> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>>>>> >> > INFORMATION IS >>>>>>>>> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>>> >> > RECIPIENT, PLEASE >>>>>>>>> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>>> >> > E-MAIL FROM YOUR >>>>>>>>> >> > SYSTEM AND DESTROY ANY COPIES. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > >>>>>>>>> > Charles Kozler >>>>>>>>> > Vice President, IT Operations >>>>>>>>> > >>>>>>>>> > FIX Flyer, LLC >>>>>>>>> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>>> > 1-888-349-3593 >>>>>>>>> > http://www.fixflyer.com >>>>>>>>> > >>>>>>>>> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>>>>>> > RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>>> > WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>>>>> > DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>>>>> > INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>>> > RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>>> > E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>>>>>>> > >>>>>>>>> > _______________________________________________ >>>>>>>>> > Users mailing list >>>>>>>>> > Users(a)ovirt.org >>>>>>>>> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Charles Kozler >>>>>>>> Vice President, IT Operations >>>>>>>> >>>>>>>> FIX Flyer, LLC >>>>>>>> 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>> 1-888-349-3593 >>>>>>>> http://www.fixflyer.com >>>>>>>> >>>>>>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sandro Bonazzola >>>>>>> Better technology. Faster innovation. Powered by community >>>>>>> collaboration. >>>>>>> See how it works at redhat.com >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Charles Kozler >>>>> Vice President, IT Operations >>>>> >>>>> FIX Flyer, LLC >>>>> 225 Broadway | Suite 1600 | New York, NY 10007 >>>>> 1-888-349-3593 >>>>> http://www.fixflyer.com >>>>> >>>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>> >>>> >>