<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 1, 2016 at 5:40 PM, Charles Kozler <span dir="ltr"><<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sandro / Nir -<div><br></div><div>I followed your steps plus </div><div><br></div><div><a href="http://www.ovirt.org/OVirt_3.6_Release_Notes#Fedora_.2F_CentOS_.2F_RHEL" target="_blank">http://www.ovirt.org/OVirt_3.6_Release_Notes#Fedora_.2F_CentOS_.2F_RHEL</a><br></div><div><br></div><div>Engine upgraded fine but then when I got to upgrading a node I did:</div><div><br></div><div>$ yum install <a href="http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm" target="_blank">http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm</a></div><div>$ yum update -y</div><div><br></div><div>And then rebooted the node. I noticed libvirt was updated by a .1 release number but vdsm (where the memory leak issue I thought was?) was not upgraded. In fact, very little of ovirt packages on the node were noticeably not updated</div><div><br></div></div></blockquote><div><br></div><div>We are not building vdsm for el6 in 3.6, you need also to upgrade to el7 if you want that.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>Updated node received the following updated packages during the install:</div><div><br></div><div><a href="http://pastebin.ca/3362714" target="_blank">http://pastebin.ca/3362714</a><br></div><div><br></div><div>Note specifically the only packages updated via the ovirt3.6 repository was ioprocess, otopi, ovirt-engine-sdk-python, ovirt-host-deploy, ovirt-release36, and python-ioprocess. I had expected to see some packages like vdsm and the likes updated - or was this not the case?</div><div><br></div><div>Upgraded node:</div><div><br></div><div><div>[compute[root@node02 yum.repos.d]$ rpm -qa | grep -i vdsm</div><div>vdsm-4.16.30-0.el6.x86_64</div><div>vdsm-python-zombiereaper-4.16.30-0.el6.noarch</div><div>vdsm-cli-4.16.30-0.el6.noarch</div><div>vdsm-yajsonrpc-4.16.30-0.el6.noarch</div><div>vdsm-jsonrpc-4.16.30-0.el6.noarch</div><div>vdsm-xmlrpc-4.16.30-0.el6.noarch</div><div>vdsm-python-4.16.30-0.el6.noarch</div></div><div><br></div><div>Nonupgraded node </div><div><br></div><div><div>[compute[root@node01 ~]$ rpm -qa | grep -i vdsm </div><div>vdsm-cli-4.16.30-0.el6.noarch</div><div>vdsm-jsonrpc-4.16.30-0.el6.noarch</div><div>vdsm-python-zombiereaper-4.16.30-0.el6.noarch</div><div>vdsm-xmlrpc-4.16.30-0.el6.noarch</div><div>vdsm-yajsonrpc-4.16.30-0.el6.noarch</div><div>vdsm-4.16.30-0.el6.x86_64</div><div>vdsm-python-4.16.30-0.el6.noarch</div></div><div><br></div><div>Also, the docs stated that the engine VM would migrate to the freshly upgraded node since it would have a higher number but it did not </div><div><br></div><div>So I cant really confirm whether or not my issue will be resolved? Or that if the node was actually updated properly?</div><div><br></div><div>Please advise on how to confirm</div><div><br></div><div>Thank you!</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 23, 2016 at 12:55 AM, Charles Kozler <span dir="ltr"><<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Thanks Sandro. Should clarify my storage is external on a redundant SAN. The steps I was concerned about was the actual upgrade. I tried to upgrade before and it brought my entire stack crumbling down so I'm hesitant. This bug seems like a huge bug that should at least somehow backported if at all possible because, to me, it renders the entire 3.5.6 branch unusable as no VMs can be deployed since OOM will eventually kill them. In any case that's just my opinion and I'm a new user to ovirt. The docs I followed originally got me going how I need and somehow didn't work for 3.6 in the same fashion so naturally I'm hesitant to upgrade but clearly have no option if I want to continue my infrastructure on ovirt. Thank you again for taking the time out to assist me, I truly appreciate it. I will try an upgrade next week and pray it all goes well :-)</p><div><div>
<div class="gmail_quote">On Jan 23, 2016 12:40 AM, "Sandro Bonazzola" <<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 22, 2016 at 10:53 PM, Charles Kozler <span dir="ltr"><<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sandro -<div><br></div><div>Do you have available documentation that can support upgrading self hosted? I followed this <a href="http://community.redhat.com/blog/2014/10/up-and-running-with-ovirt-3-5/" target="_blank">http://community.redhat.com/blog/2014/10/up-and-running-with-ovirt-3-5/</a></div><div><br></div><div>Would it be as easy as installing the RPM and then running yum upgrade?</div><div><br></div></div></blockquote><br>Note that mentioned article describes an unsupported hyperconverged setup running NFS over Gluster.<br>That said,<br>1) put the hosted-engine storage domain into global maintenance mode<br>2) upgrade the engine VM<br>3) select the first host to upgrade and put it under maintenance from the engine, wait for the engine vm to migrate if needed.<br>4) yum upgrade the first host and wait until ovirt-ha-agent completes<br>5) exit global and local maintenance mode<br>6) repeat 3-5 on all the other hosts<br>7) once all hosts are updated you can increase the cluster compatibility level to 3.6. At this point the engine will trigger the auto-import of the hosted-engine storage domain.<div><br></div><div>Simone, Roy, can you confirm above steps? Maybe also you can update <a href="http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine" target="_blank">http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine</a></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Thanks</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 22, 2016 at 4:42 PM, Sandro Bonazzola <span dir="ltr"><<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><p dir="ltr"><br>
Il 22/Gen/2016 22:31, "Charles Kozler" <<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>> ha scritto:<br>
><br>
> Hi Nir -<br>
><br>
> do you have a release target date for 3.5.8? Any estimate would help.<br>
><br></p>
</span><p dir="ltr">There won't be any supported release after 3.5.6. Please update to 3.6.2 next week <br><br></p><div><div>
<p dir="ltr">> If its not VDSM, what is it exactly? Sorry, I understood from the ticket it was something inside vdsm, was I mistaken?<br>
><br>
> CentOS 6 is the servers. 6.7 to be exact<br>
><br>
> I have done all forms of flushing that I can (page cache, inodes, dentry's, etc) and as well moved VM's around to other nodes and nothing changes the memory. How can I find the leak? Where is the leak? RES shows the following of which, the totals dont add up to 20GB<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
> 19044 qemu 20 0 8876m 4.0g 5680 S 3.6 12.9 1571:44 qemu-kvm <br>
> 26143 qemu 20 0 5094m 1.1g 5624 S 9.2 3.7 6012:12 qemu-kvm <br>
> 5837 root 0 -20 964m 624m 3664 S 0.0 2.0 85:22.09 glusterfs <br>
> 14328 root 0 -20 635m 169m 3384 S 0.0 0.5 43:15.23 glusterfs <br>
> 5134 vdsm 0 -20 4368m 111m 10m S 5.9 0.3 3710:50 vdsm <br>
> 4095 root 15 -5 727m 43m 10m S 0.0 0.1 0:02.00 supervdsmServer <br>
><br>
> 4.0G + 1.1G + 624M + 169 + 111M + 43M = ~7GB<br>
><br>
> This was top sorted by RES from highest to lowest<br>
><br>
> At that point I wouldnt know where else to look except slab / kernel structures. Of which slab shows:<br>
><br>
> [compute[root@node1 ~]$ cat /proc/meminfo | grep -i slab<br>
> Slab: 2549748 kB<br>
><br>
> So roughly 2-3GB. Adding that to the other use of 7GB we have still about 10GB unaccounted for<br>
><br>
> On Fri, Jan 22, 2016 at 4:24 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>> wrote:<br>
>><br>
>> On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler <<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>> wrote:<br>
>> > Hi Nir -<br>
>> ><br>
>> > Thanks for getting back to me. Will the patch to 3.6 be backported to 3.5?<br>
>><br>
>> We plan to include them in 3.5.8.<br>
>><br>
>> > As you can tell from the images, it takes days and days for it to increase<br>
>> > over time. I also wasnt sure if that was the right bug because VDSM memory<br>
>> > shows normal from top ...<br>
>> ><br>
>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
>> > 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28 vdsm<br>
>><br>
>> As you wrote, this issue is not related to vdsm.<br>
>><br>
>> ><br>
>> > Res is only 111M. This is from node1 which is showing currently 20GB of 32GB<br>
>> > used with only 2 VMs running on it - 1 with 4G and another with ~1 GB of RAM<br>
>> > configured<br>
>> ><br>
>> > The images are from nagios and the value here is a direct correlation to<br>
>> > what you would see in the free command output. See below from an example of<br>
>> > node 1 and node 2<br>
>> ><br>
>> > [compute[root@node1 ~]$ free<br>
>> > total used free shared buffers cached<br>
>> > Mem: 32765316 20318156 12447160 252 30884 628948<br>
>> > -/+ buffers/cache: 19658324 <a href="tel:13106992" value="+13106992" target="_blank">13106992</a><br>
>> > Swap: 19247100 0 19247100<br>
>> > [compute[root@node1 ~]$ free -m<br>
>> > total used free shared buffers cached<br>
>> > Mem: 31997 19843 12153 0 30 614<br>
>> > -/+ buffers/cache: 19199 12798<br>
>> > Swap: 18795 0 18795<br>
>> ><br>
>> > And its correlated image <a href="http://i.imgur.com/PZLEgyx.png" target="_blank">http://i.imgur.com/PZLEgyx.png</a> (~19GB used)<br>
>> ><br>
>> > And as a control, node 2 that I just restarted today<br>
>> ><br>
>> > [compute[root@node2 ~]$ free<br>
>> > total used free shared buffers cached<br>
>> > Mem: 32765316 1815324 30949992 212 35784 717320<br>
>> > -/+ buffers/cache: 1062220 31703096<br>
>> > Swap: 19247100 0 19247100<br>
>><br>
>> Is this rhel/centos 6?<br>
>><br>
>> > [compute[root@node2 ~]$ free -m<br>
>> > total used free shared buffers cached<br>
>> > Mem: 31997 1772 30225 0 34 700<br>
>> > -/+ buffers/cache: 1036 30960<br>
>> > Swap: 18795 0 18795<br>
>> ><br>
>> > And its correlated image <a href="http://i.imgur.com/8ldPVqY.png" target="_blank">http://i.imgur.com/8ldPVqY.png</a> (~2GB used). Note<br>
>> > how 1772 in the image is exactly what is registered under 'used' in free<br>
>> > command<br>
>><br>
>> I guess you should start looking at the processes running on these nodes.<br>
>><br>
>> Maybe try to collect memory usage per process using ps?<br>
>><br>
>> ><br>
>> > On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>> wrote:<br>
>> >><br>
>> >> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler <<a href="mailto:charles@fixflyer.com" target="_blank">charles@fixflyer.com</a>><br>
>> >> wrote:<br>
>> >> > Here is a screenshot of my three nodes and their increased memory usage<br>
>> >> > over<br>
>> >> > 30 days. Note that node #2 had 1 single VM that had 4GB of RAM assigned<br>
>> >> > to<br>
>> >> > it. I had since shut it down and saw no memory reclamation occur.<br>
>> >> > Further, I<br>
>> >> > flushed page caches and inodes and ran 'sync'. I tried everything but<br>
>> >> > nothing brought the memory usage down. vdsm was low too (couple hundred<br>
>> >> > MB)<br>
>> >><br>
>> >> Note that there is an old leak in vdsm, will be fixed in next 3.6 build:<br>
>> >> <a href="https://bugzilla.redhat.com/1269424" target="_blank">https://bugzilla.redhat.com/1269424</a><br>
>> >><br>
>> >> > and there was no qemu-kvm process running so I'm at a loss<br>
>> >> ><br>
>> >> > <a href="http://imgur.com/a/aFPcK" target="_blank">http://imgur.com/a/aFPcK</a><br>
>> >> ><br>
>> >> > Please advise on what I can do to debug this. Note I have restarted node<br>
>> >> > 2<br>
>> >> > (which is why you see the drop) to see if it raises in memory use over<br>
>> >> > tim<br>
>> >> > even with no VM's running<br>
>> >><br>
>> >> Not sure what is "memory" that you show in the graphs. Theoretically this<br>
>> >> may be<br>
>> >> normal memory usage, Linux using free memory for the buffer cache.<br>
>> >><br>
>> >> Can you instead show the output of "free", during one day, maybe run once<br>
>> >> per hour?<br>
>> >><br>
>> >> You may also like to install sysstat for collecting and monitoring<br>
>> >> resources usage.<br>
>> >><br>
>> >> ><br>
>> >> > [compute[root@node2 log]$ rpm -qa | grep -i ovirt<br>
>> >> > libgovirt-0.3.2-1.el6.x86_64<br>
>> >> > ovirt-release35-006-1.noarch<br>
>> >> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch<br>
>> >> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch<br>
>> >> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch<br>
>> >> > ovirt-host-deploy-1.3.2-1.el6.noarch<br>
>> >> ><br>
>> >> ><br>
>> >> > --<br>
>> >> ><br>
>> >> > Charles Kozler<br>
>> >> > Vice President, IT Operations<br>
>> >> ><br>
>> >> > FIX Flyer, LLC<br>
>> >> > 225 Broadway | Suite 1600 | New York, NY 10007<br>
>> >> > <a href="tel:1-888-349-3593" value="+18883493593" target="_blank">1-888-349-3593</a><br>
>> >> > <a href="http://www.fixflyer.com" target="_blank">http://www.fixflyer.com</a><br>
>> >> ><br>
>> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED<br>
>> >> > RECIPIENT(S)<br>
>> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS<br>
>> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING,<br>
>> >> > DISTRIBUTION,<br>
>> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION<br>
>> >> > IS<br>
>> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT,<br>
>> >> > PLEASE<br>
>> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM<br>
>> >> > YOUR<br>
>> >> > SYSTEM AND DESTROY ANY COPIES.<br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > Users mailing list<br>
>> >> > <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
>> >> > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>> >> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> ><br>
>> > Charles Kozler<br>
>> > Vice President, IT Operations<br>
>> ><br>
>> > FIX Flyer, LLC<br>
>> > 225 Broadway | Suite 1600 | New York, NY 10007<br>
>> > <a href="tel:1-888-349-3593" value="+18883493593" target="_blank">1-888-349-3593</a><br>
>> > <a href="http://www.fixflyer.com" target="_blank">http://www.fixflyer.com</a><br>
>> ><br>
>> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S)<br>
>> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS<br>
>> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION,<br>
>> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS<br>
>> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE<br>
>> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR<br>
>> > SYSTEM AND DESTROY ANY COPIES.<br>
><br>
><br>
><br>
><br>
> -- <br>
><br>
> Charles Kozler<br>
> Vice President, IT Operations<br>
> <br>
> FIX Flyer, LLC<br>
> 225 Broadway | Suite 1600 | New York, NY 10007<br>
> <a href="tel:1-888-349-3593" value="+18883493593" target="_blank">1-888-349-3593</a><br>
> <a href="http://www.fixflyer.com" target="_blank">http://www.fixflyer.com</a><br>
> <br>
> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br>
</p>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:medium"><div style="color:rgb(0,0,0);font-family:"Times New Roman""><b><span style="font-family:"times new roman",times,serif"><font size="2">Charles Kozler</font></span></b></div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><i><span style="font-family:"times new roman",times,serif"><font size="2">Vice President, IT Operations</font></span></i></div><div style="color:rgb(0,0,0);font-family:"Times New Roman";font-size:medium"> </div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><span style="font-family:"times new roman",times,serif"><font size="2">FIX Flyer, LLC</font></span></div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><span style="font-family:"times new roman",times,serif"><font size="2">225 Broadway | Suite 1600 | New York, NY 10007</font></span></div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><span style="font-family:"times new roman",times,serif"><font size="2"><a href="tel:1-888-349-3593" value="+18883493593" target="_blank">1-888-349-3593</a></font></span></div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><a href="http://fixflyer.com" target="_blank"><span style="font-family:"times new roman",times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><font size="2"> </font></div><div style="color:rgb(0,0,0);font-family:"Times New Roman""><font size="1">NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.</font></div></div></div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div><div dir="ltr"><div><div dir="ltr">Sandro Bonazzola<br>Better technology. Faster innovation. Powered by community collaboration.<br>See how it works at <a href="http://redhat.com" target="_blank">redhat.com</a><br></div></div></div></div>
</div></div>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium"><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><strong><span style="font-family:'times new roman',times,serif"><font size="2">Charles Kozler</font></span></strong></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><em><span style="font-family:'times new roman',times,serif"><font size="2">Vice President, IT Operations</font></span></em></div><div style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium"> </div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">FIX Flyer, LLC</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">225 Broadway | Suite 1600 | New York, NY 10007</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2"><a href="tel:1-888-349-3593" value="+18883493593" target="_blank">1-888-349-3593</a></font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><a href="http://fixflyer.com" target="_blank"><span style="font-family:'times new roman',times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><font size="2"> </font></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><font size="1">NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.</font></div></div></div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br></div></div>