Re: [ovirt-users] memory leak in 3.5.6 - not vdsm

So I will still have the memory leak? On Feb 1, 2016 12:39 PM, "Simone Tiraboschi" <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Feb 1, 2016 at 6:33 PM, Charles Kozler <charles(a)fixflyer.com&gt; > wrote: > >> So what about the bug that I hit for vdsm as listed above by Nir? Will I >> have that patch to avoid the memory leak or no? Upgrading an entire node to >> centos 7 is not actually feasible and was previously outlined above that I >> just needed to upgrade to ovirt 3.6 and no mention of OS change ... >> > > You cannot install VDSM from 3.6 on el6: > > http://www.ovirt.org/OVirt_3.6_Release_Notes#RHEL_6.7_-_CentOS_6.7_and_si... > and there is no plan for a 3.5.8. > > > >> On Feb 1, 2016 12:30 PM, "Simone Tiraboschi" <stirabos(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Mon, Feb 1, 2016 at 5:40 PM, Charles Kozler <charles(a)fixflyer.com&gt; >>> wrote: >>> >>>> Sandro / Nir - >>>> >>>> I followed your steps plus >>>> >>>> http://www.ovirt.org/OVirt_3.6_Release_Notes#Fedora_.2F_CentOS_.2F_RHEL >>>> >>>> Engine upgraded fine but then when I got to upgrading a node I did: >>>> >>>> $ yum install >>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm >>>> $ yum update -y >>>> >>>> And then rebooted the node. I noticed libvirt was updated by a .1 >>>> release number but vdsm (where the memory leak issue I thought was?) was >>>> not upgraded. In fact, very little of ovirt packages on the node were >>>> noticeably not updated >>>> >>>> >>> We are not building vdsm for el6 in 3.6, you need also to upgrade to >>> el7 if you want that. >>> >>> >>>> Updated node received the following updated packages during the >>>> install: >>>> >>>> http://pastebin.ca/3362714 >>>> >>>> Note specifically the only packages updated via the ovirt3.6 >>>> repository >>>> was ioprocess, otopi, ovirt-engine-sdk-python, ovirt-host-deploy, >>>> ovirt-release36, and python-ioprocess. I had expected to see some packages >>>> like vdsm and the likes updated - or was this not the case? >>>> >>>> Upgraded node: >>>> >>>> [compute[root@node02 yum.repos.d]$ rpm -qa | grep -i vdsm >>>> vdsm-4.16.30-0.el6.x86_64 >>>> vdsm-python-zombiereaper-4.16.30-0.el6.noarch >>>> vdsm-cli-4.16.30-0.el6.noarch >>>> vdsm-yajsonrpc-4.16.30-0.el6.noarch >>>> vdsm-jsonrpc-4.16.30-0.el6.noarch >>>> vdsm-xmlrpc-4.16.30-0.el6.noarch >>>> vdsm-python-4.16.30-0.el6.noarch >>>> >>>> Nonupgraded node >>>> >>>> [compute[root@node01 ~]$ rpm -qa | grep -i vdsm >>>> vdsm-cli-4.16.30-0.el6.noarch >>>> vdsm-jsonrpc-4.16.30-0.el6.noarch >>>> vdsm-python-zombiereaper-4.16.30-0.el6.noarch >>>> vdsm-xmlrpc-4.16.30-0.el6.noarch >>>> vdsm-yajsonrpc-4.16.30-0.el6.noarch >>>> vdsm-4.16.30-0.el6.x86_64 >>>> vdsm-python-4.16.30-0.el6.noarch >>>> >>>> Also, the docs stated that the engine VM would migrate to the freshly >>>> upgraded node since it would have a higher number but it did not >>>> >>>> So I cant really confirm whether or not my issue will be resolved? Or >>>> that if the node was actually updated properly? >>>> >>>> Please advise on how to confirm >>>> >>>> Thank you! >>>> >>>> On Sat, Jan 23, 2016 at 12:55 AM, Charles Kozler <charles(a)fixflyer.com >>>> > wrote: >>>> >>>>> Thanks Sandro. Should clarify my storage is external on a redundant >>>>> SAN. The steps I was concerned about was the actual upgrade. I tried to >>>>> upgrade before and it brought my entire stack crumbling down so I'm >>>>> hesitant. This bug seems like a huge bug that should at least somehow >>>>> backported if at all possible because, to me, it renders the entire 3.5.6 >>>>> branch unusable as no VMs can be deployed since OOM will eventually kill >>>>> them. In any case that's just my opinion and I'm a new user to ovirt. The >>>>> docs I followed originally got me going how I need and somehow didn't work >>>>> for 3.6 in the same fashion so naturally I'm hesitant to upgrade but >>>>> clearly have no option if I want to continue my infrastructure on ovirt. >>>>> Thank you again for taking the time out to assist me, I truly appreciate >>>>> it. I will try an upgrade next week and pray it all goes well :-) >>>>> On Jan 23, 2016 12:40 AM, "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 22, 2016 at 10:53 PM, Charles Kozler < >>>>>> charles(a)fixflyer.com&gt; wrote: >>>>>> >>>>>>> Sandro - >>>>>>> >>>>>>> Do you have available documentation that can support upgrading self >>>>>>> hosted? I followed this >>>>>>> http://community.redhat.com/blog/2014/10/up-and-running-with-ovirt-3-5/ >>>>>>> >>>>>>> Would it be as easy as installing the RPM and then running yum >>>>>>> upgrade? >>>>>>> >>>>>>> >>>>>> Note that mentioned article describes an unsupported hyperconverged >>>>>> setup running NFS over Gluster. >>>>>> That said, >>>>>> 1) put the hosted-engine storage domain into global maintenance mode >>>>>> 2) upgrade the engine VM >>>>>> 3) select the first host to upgrade and put it under maintenance >>>>>> from the engine, wait for the engine vm to migrate if needed. >>>>>> 4) yum upgrade the first host and wait until ovirt-ha-agent completes >>>>>> 5) exit global and local maintenance mode >>>>>> 6) repeat 3-5 on all the other hosts >>>>>> 7) once all hosts are updated you can increase the cluster >>>>>> compatibility level to 3.6. At this point the engine will trigger the >>>>>> auto-import of the hosted-engine storage domain. >>>>>> >>>>>> Simone, Roy, can you confirm above steps? Maybe also you can update >>>>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine >>>>>> >>>>>> >>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Fri, Jan 22, 2016 at 4:42 PM, Sandro Bonazzola < >>>>>>> sbonazzo(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> >>>>>>>> Il 22/Gen/2016 22:31, "Charles Kozler" <charles(a)fixflyer.com&gt; ha >>>>>>>> scritto: >>>>>>>> > >>>>>>>> > Hi Nir - >>>>>>>> > >>>>>>>> > do you have a release target date for 3.5.8? Any estimate would >>>>>>>> help. >>>>>>>> > >>>>>>>> >>>>>>>> There won't be any supported release after 3.5.6. Please update to >>>>>>>> 3.6.2 next week >>>>>>>> >>>>>>>> > If its not VDSM, what is it exactly? Sorry, I understood from >>>>>>>> the ticket it was something inside vdsm, was I mistaken? >>>>>>>> > >>>>>>>> > CentOS 6 is the servers. 6.7 to be exact >>>>>>>> > >>>>>>>> > I have done all forms of flushing that I can (page cache, >>>>>>>> inodes, dentry's, etc) and as well moved VM's around to other nodes and >>>>>>>> nothing changes the memory. How can I find the leak? Where is the leak? RES >>>>>>>> shows the following of which, the totals dont add up to 20GB >>>>>>>> > >>>>>>>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>>> COMMAND >>>>>>>> >>>>>>>> > 19044 qemu 20 0 8876m 4.0g 5680 S 3.6 12.9 1571:44 >>>>>>>> qemu-kvm >>>>>>>> >>>>>>>> > 26143 qemu 20 0 5094m 1.1g 5624 S 9.2 3.7 6012:12 >>>>>>>> qemu-kvm >>>>>>>> >>>>>>>> > 5837 root 0 -20 964m 624m 3664 S 0.0 2.0 85:22.09 >>>>>>>> glusterfs >>>>>>>> >>>>>>>> > 14328 root 0 -20 635m 169m 3384 S 0.0 0.5 43:15.23 >>>>>>>> glusterfs >>>>>>>> >>>>>>>> > 5134 vdsm 0 -20 4368m 111m 10m S 5.9 0.3 3710:50 >>>>>>>> vdsm >>>>>>>> >>>>>>>> > 4095 root 15 -5 727m 43m 10m S 0.0 0.1 0:02.00 >>>>>>>> supervdsmServer >>>>>>>> > >>>>>>>> > 4.0G + 1.1G + 624M + 169 + 111M + 43M = ~7GB >>>>>>>> > >>>>>>>> > This was top sorted by RES from highest to lowest >>>>>>>> > >>>>>>>> > At that point I wouldnt know where else to look except slab / >>>>>>>> kernel structures. Of which slab shows: >>>>>>>> > >>>>>>>> > [compute[root@node1 ~]$ cat /proc/meminfo | grep -i slab >>>>>>>> > Slab: 2549748 kB >>>>>>>> > >>>>>>>> > So roughly 2-3GB. Adding that to the other use of 7GB we have >>>>>>>> still about 10GB unaccounted for >>>>>>>> > >>>>>>>> > On Fri, Jan 22, 2016 at 4:24 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler < >>>>>>>> charles(a)fixflyer.com&gt; wrote: >>>>>>>> >> > Hi Nir - >>>>>>>> >> > >>>>>>>> >> > Thanks for getting back to me. Will the patch to 3.6 be >>>>>>>> backported to 3.5? >>>>>>>> >> >>>>>>>> >> We plan to include them in 3.5.8. >>>>>>>> >> >>>>>>>> >> > As you can tell from the images, it takes days and days for >>>>>>>> it to increase >>>>>>>> >> > over time. I also wasnt sure if that was the right bug >>>>>>>> because VDSM memory >>>>>>>> >> > shows normal from top ... >>>>>>>> >> > >>>>>>>> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>>>> COMMAND >>>>>>>> >> > 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28 >>>>>>>> vdsm >>>>>>>> >> >>>>>>>> >> As you wrote, this issue is not related to vdsm. >>>>>>>> >> >>>>>>>> >> > >>>>>>>> >> > Res is only 111M. This is from node1 which is showing >>>>>>>> currently 20GB of 32GB >>>>>>>> >> > used with only 2 VMs running on it - 1 with 4G and another >>>>>>>> with ~1 GB of RAM >>>>>>>> >> > configured >>>>>>>> >> > >>>>>>>> >> > The images are from nagios and the value here is a direct >>>>>>>> correlation to >>>>>>>> >> > what you would see in the free command output. See below from >>>>>>>> an example of >>>>>>>> >> > node 1 and node 2 >>>>>>>> >> > >>>>>>>> >> > [compute[root@node1 ~]$ free >>>>>>>> >> > total used free shared >>>>>>>> buffers cached >>>>>>>> >> > Mem: 32765316 20318156 12447160 252 >>>>>>>> 30884 628948 >>>>>>>> >> > -/+ buffers/cache: 19658324 13106992 >>>>>>>> >> > Swap: 19247100 0 19247100 >>>>>>>> >> > [compute[root@node1 ~]$ free -m >>>>>>>> >> > total used free shared >>>>>>>> buffers cached >>>>>>>> >> > Mem: 31997 19843 12153 0 >>>>>>>> 30 614 >>>>>>>> >> > -/+ buffers/cache: 19199 12798 >>>>>>>> >> > Swap: 18795 0 18795 >>>>>>>> >> > >>>>>>>> >> > And its correlated image http://i.imgur.com/PZLEgyx.png >>>>>>>> (~19GB used) >>>>>>>> >> > >>>>>>>> >> > And as a control, node 2 that I just restarted today >>>>>>>> >> > >>>>>>>> >> > [compute[root@node2 ~]$ free >>>>>>>> >> > total used free shared >>>>>>>> buffers cached >>>>>>>> >> > Mem: 32765316 1815324 30949992 212 >>>>>>>> 35784 717320 >>>>>>>> >> > -/+ buffers/cache: 1062220 31703096 >>>>>>>> >> > Swap: 19247100 0 19247100 >>>>>>>> >> >>>>>>>> >> Is this rhel/centos 6? >>>>>>>> >> >>>>>>>> >> > [compute[root@node2 ~]$ free -m >>>>>>>> >> > total used free shared >>>>>>>> buffers cached >>>>>>>> >> > Mem: 31997 1772 30225 0 >>>>>>>> 34 700 >>>>>>>> >> > -/+ buffers/cache: 1036 30960 >>>>>>>> >> > Swap: 18795 0 18795 >>>>>>>> >> > >>>>>>>> >> > And its correlated image http://i.imgur.com/8ldPVqY.png >>>>>>>> (~2GB used). Note >>>>>>>> >> > how 1772 in the image is exactly what is registered under >>>>>>>> 'used' in free >>>>>>>> >> > command >>>>>>>> >> >>>>>>>> >> I guess you should start looking at the processes running on >>>>>>>> these nodes. >>>>>>>> >> >>>>>>>> >> Maybe try to collect memory usage per process using ps? >>>>>>>> >> >>>>>>>> >> > >>>>>>>> >> > On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer < >>>>>>>> nsoffer(a)redhat.com&gt; wrote: >>>>>>>> >> >> >>>>>>>> >> >> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler < >>>>>>>> charles(a)fixflyer.com&gt; >>>>>>>> >> >> wrote: >>>>>>>> >> >> > Here is a screenshot of my three nodes and their increased >>>>>>>> memory usage >>>>>>>> >> >> > over >>>>>>>> >> >> > 30 days. Note that node #2 had 1 single VM that had 4GB of >>>>>>>> RAM assigned >>>>>>>> >> >> > to >>>>>>>> >> >> > it. I had since shut it down and saw no memory reclamation >>>>>>>> occur. >>>>>>>> >> >> > Further, I >>>>>>>> >> >> > flushed page caches and inodes and ran 'sync'. I tried >>>>>>>> everything but >>>>>>>> >> >> > nothing brought the memory usage down. vdsm was low too >>>>>>>> (couple hundred >>>>>>>> >> >> > MB) >>>>>>>> >> >> >>>>>>>> >> >> Note that there is an old leak in vdsm, will be fixed in >>>>>>>> next 3.6 build: >>>>>>>> >> >> https://bugzilla.redhat.com/1269424 >>>>>>>> >> >> >>>>>>>> >> >> > and there was no qemu-kvm process running so I'm at a loss >>>>>>>> >> >> > >>>>>>>> >> >> > http://imgur.com/a/aFPcK >>>>>>>> >> >> > >>>>>>>> >> >> > Please advise on what I can do to debug this. Note I have >>>>>>>> restarted node >>>>>>>> >> >> > 2 >>>>>>>> >> >> > (which is why you see the drop) to see if it raises in >>>>>>>> memory use over >>>>>>>> >> >> > tim >>>>>>>> >> >> > even with no VM's running >>>>>>>> >> >> >>>>>>>> >> >> Not sure what is "memory" that you show in the graphs. >>>>>>>> Theoretically this >>>>>>>> >> >> may be >>>>>>>> >> >> normal memory usage, Linux using free memory for the buffer >>>>>>>> cache. >>>>>>>> >> >> >>>>>>>> >> >> Can you instead show the output of "free", during one day, >>>>>>>> maybe run once >>>>>>>> >> >> per hour? >>>>>>>> >> >> >>>>>>>> >> >> You may also like to install sysstat for collecting and >>>>>>>> monitoring >>>>>>>> >> >> resources usage. >>>>>>>> >> >> >>>>>>>> >> >> > >>>>>>>> >> >> > [compute[root@node2 log]$ rpm -qa | grep -i ovirt >>>>>>>> >> >> > libgovirt-0.3.2-1.el6.x86_64 >>>>>>>> >> >> > ovirt-release35-006-1.noarch >>>>>>>> >> >> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch >>>>>>>> >> >> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch >>>>>>>> >> >> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch >>>>>>>> >> >> > ovirt-host-deploy-1.3.2-1.el6.noarch >>>>>>>> >> >> > >>>>>>>> >> >> > >>>>>>>> >> >> > -- >>>>>>>> >> >> > >>>>>>>> >> >> > Charles Kozler >>>>>>>> >> >> > Vice President, IT Operations >>>>>>>> >> >> > >>>>>>>> >> >> > FIX Flyer, LLC >>>>>>>> >> >> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>> >> >> > 1-888-349-3593 >>>>>>>> >> >> > http://www.fixflyer.com >>>>>>>> >> >> > >>>>>>>> >> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE >>>>>>>> INTENDED >>>>>>>> >> >> > RECIPIENT(S) >>>>>>>> >> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>> WHICH IS >>>>>>>> >> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, >>>>>>>> COPYING, >>>>>>>> >> >> > DISTRIBUTION, >>>>>>>> >> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO >>>>>>>> THIS INFORMATION >>>>>>>> >> >> > IS >>>>>>>> >> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>> RECIPIENT, >>>>>>>> >> >> > PLEASE >>>>>>>> >> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>> E-MAIL FROM >>>>>>>> >> >> > YOUR >>>>>>>> >> >> > SYSTEM AND DESTROY ANY COPIES. >>>>>>>> >> >> > >>>>>>>> >> >> > _______________________________________________ >>>>>>>> >> >> > Users mailing list >>>>>>>> >> >> > Users(a)ovirt.org >>>>>>>> >> >> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > -- >>>>>>>> >> > >>>>>>>> >> > Charles Kozler >>>>>>>> >> > Vice President, IT Operations >>>>>>>> >> > >>>>>>>> >> > FIX Flyer, LLC >>>>>>>> >> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>> >> > 1-888-349-3593 >>>>>>>> >> > http://www.fixflyer.com >>>>>>>> >> > >>>>>>>> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE >>>>>>>> INTENDED RECIPIENT(S) >>>>>>>> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>> WHICH IS >>>>>>>> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>>>> DISTRIBUTION, >>>>>>>> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>>>> INFORMATION IS >>>>>>>> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>> RECIPIENT, PLEASE >>>>>>>> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>> E-MAIL FROM YOUR >>>>>>>> >> > SYSTEM AND DESTROY ANY COPIES. >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > -- >>>>>>>> > >>>>>>>> > Charles Kozler >>>>>>>> > Vice President, IT Operations >>>>>>>> > >>>>>>>> > FIX Flyer, LLC >>>>>>>> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>>> > 1-888-349-3593 >>>>>>>> > http://www.fixflyer.com >>>>>>>> > >>>>>>>> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > Users mailing list >>>>>>>> > Users(a)ovirt.org >>>>>>>> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *Charles Kozler* >>>>>>> *Vice President, IT Operations* >>>>>>> >>>>>>> FIX Flyer, LLC >>>>>>> 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>>> 1-888-349-3593 >>>>>>> http://www.fixflyer.com <http://fixflyer.com> >>>>>>> >>>>>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sandro Bonazzola >>>>>> Better technology. Faster innovation. Powered by community >>>>>> collaboration. >>>>>> See how it works at redhat.com >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *Charles Kozler* >>>> *Vice President, IT Operations* >>>> >>>> FIX Flyer, LLC >>>> 225 Broadway | Suite 1600 | New York, NY 10007 >>>> 1-888-349-3593 >>>> http://www.fixflyer.com <http://fixflyer.com> >>>> >>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>> >>> >>> >