<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:black;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Hi Michael,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Thanks for the additional steps and troubleshooting, things are looking healthier, but the virsh config has not changed. I can see a
direct-pool configuration which looks correct.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Did it fail to clean up the vdsm-SRIOV interface & associated bridge? Or should they be present as well? I did a reboot of the host and
it looks like it sets them all up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US"># virsh -r net-list --all<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">Name State Autostart Persistent<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">----------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">;vdsmdummy; active no no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">direct-pool active yes yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">vdsm-ovirtmgmt active yes yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">vdsm-SRIOV active yes yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">$ brctl show<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">bridge name bridge id STP enabled interfaces<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">;vdsmdummy; 8000.000000000000 no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">SRIOV 8000.0025b5000b0f no enp7s0f0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-US">ovirtmgmt 8000.0025b5000b4f no enp6s0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"># virsh -r net-dumpxml vdsm-SRIOV<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><network><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> <name>vdsm-SRIOV</name><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> <uuid>46b2655c-ce6f-401d-acaf-f5ca9cdb1627</uuid><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> <forward mode='bridge'/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> <bridge name='SRIOV'/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"></network><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"># rpm -qa | grep -i vmfex<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">vdsm-hook-vmfex-dev-4.17.23-1.el7.noarch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Found your bug report - is this related as well? As it should be resolved in the version I’m using. https://bugzilla.redhat.com/show_bug.cgi?id=1308839<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><img width="1001" height="23" id="Picture_x0020_1" src="cid:image002.png@01D17B0F.3AE61C60"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Thread-196::ERROR::2016-03-10 20:47:26,566::vm::759::virt.vm::(_startUnderlyingVm) vmId=`2a03a747-5d43-4ba9-a95b-6903fb41cacc`::The vm
start process failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Traceback (most recent call last):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> self._run()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> File "/usr/share/vdsm/virt/vm.py", line 1941, in _run<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> self._connection.createXML(domxml, flags),<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> ret = f(*args, **kwargs)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> return func(inst, *args, **kwargs)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in createXML<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"> if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">libvirtError: unsupported configuration: filterref is not supported for network interfaces of type direct<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Thread-196::INFO::2016-03-10 20:47:26,567::vm::1330::virt.vm::(setDownStatus) vmId=`2a03a747-5d43-4ba9-a95b-6903fb41cacc`::Changed state
to Down: unsupported configuration: filterref is not supported for network interfaces of type direct (code=1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Thread-196::DEBUG::2016-03-10 20:47:26,568::vmchannels::229::vds::(unregister) Delete fileno 35 from listener.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Thread-196::DEBUG::2016-03-10 20:47:26,568::vmchannels::59::vds::(_unregister_fd) Failed to unregister FD from epoll (ENOENT): 35<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Thread-196::DEBUG::2016-03-10 20:47:26,569::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"2a03a747-5d43-4ba9-a95b-6903fb41cacc":
{"status": "Down", "timeOffset": "0", "exitReason": 1, "exitMessage": "unsupported configuration: filterref is not supported for network interfaces of type direct", "exitCode": 1}, "notify_time": 4406685340}, "jsonrpc": "2.0", "method": "|virt|VM_status|2a03a747-5d43-4ba9-a95b-6903fb41cacc"}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">VM Channels Listener::DEBUG::2016-03-10 20:47:26,780::vmchannels::135::vds::(_do_del_channels) fileno 35 was removed from listener.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">periodic/0::WARNING::2016-03-10 20:47:27,877::periodic::258::virt.periodic.VmDispatcher::(__call__) could not run <class 'virt.periodic.DriveWatermarkMonitor'>
on [u'2a03a747-5d43-4ba9-a95b-6903fb41cacc']<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">mailbox.SPMMonitor::DEBUG::2016-03-10 20:47:27,900::storage_mailbox::735::Storage.Misc.excCmd::(_checkForMail) /usr/bin/taskset --cpu-list
0-39 dd if=/rhev/data-center/00000001-0001-0001-0001-00000000017a/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">mailbox.SPMMonitor::DEBUG::2016-03-10 20:47:27,919::storage_mailbox::735::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0
records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.00310821 s, 329 MB/s\n'; <rc> = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">jsonrpc.Executor/7::DEBUG::2016-03-10 20:47:28,411::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.destroy' in bridge
with {u'vmID': u'2a03a747-5d43-4ba9-a95b-6903fb41cacc'}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">jsonrpc.Executor/7::INFO::2016-03-10 20:47:28,412::API::341::vds::(destroy) vmContainerLock acquired by vm 2a03a747-5d43-4ba9-a95b-6903fb41cacc<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">jsonrpc.Executor/7::DEBUG::2016-03-10 20:47:28,412::vm::3885::virt.vm::(destroy) vmId=`2a03a747-5d43-4ba9-a95b-6903fb41cacc`::destroy
Called<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">jsonrpc.Executor/7::INFO::2016-03-10 20:47:28,412::vm::3815::virt.vm::(releaseVm) vmId=`2a03a747-5d43-4ba9-a95b-6903fb41cacc`::Release
VM resources<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US">David<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Michael Burman [mailto:mburman@redhat.com]
<br>
<b>Sent:</b> Thursday, March 10, 2016 19:07<br>
<b>To:</b> David LeVene <David.LeVene@blackboard.com><br>
<b>Cc:</b> users@ovirt.org; Dan Kenigsberg <danken@redhat.com><br>
<b>Subject:</b> Re: [ovirt-users] Configuring the SRIOV virsh device<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Ok, lets try few things. <o:p></o:p></p>
</div>
<p class="MsoNormal">- I see you using the latest vdsm 3.6 version and you have the
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">vdsm-hook-vmfex-dev-4.17.23-1.el7.noarch shipped by default. But i also see that you have the old hook installed - vdsm-hook-vmfex-4.17.23-0.el7.centos.noarch, please remove it from
the server and re-install the host in ovirt(maintenance+re-install). To make sure you have no conflicts with the old hook and you are using the new one.
</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><br>
- In the VM interface profile dialog please uncheck the 'passthrough' property(first remove the SRIOV network from the VMs vNIC and add it back after editing the profile). The 'passthrough' property shouldn't be checked when using the vmfex-hook.
</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">- Last thing, but not must, go to 'Networks' main tab and choose your network 'SRIOV', go to 'Clusters' sub tab and press the 'Manage network' button --> uncheck
the 'Required' checkboxes</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> from your cluster(because your network attached to 2 servers in your cluster from 3, it is in down state colored with red)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">, once you will uncheck the 'Required' from the cluster it will be shown as UP, colored in green.
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">I believe now it will work for you, please let me know.
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Cheers )</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Mar 10, 2016 at 1:51 AM, David LeVene <<a href="mailto:David.LeVene@blackboard.com" target="_blank">David.LeVene@blackboard.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Hi Michael,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">I’m going to include some screen shots - as I’ve got it all setup exactly how you mention below.. but
the config on the Hypervisor is still bridged config.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">All the UCS port profiles are correct as we use this config on standalone KVM/QEMU hosts.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">See in-line for pics at the appropriate step .</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Cheers</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">David</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Michael
Burman [mailto:<a href="mailto:mburman@redhat.com" target="_blank">mburman@redhat.com</a>]
<br>
<b>Sent:</b> Wednesday, March 09, 2016 19:47<br>
<b>To:</b> David LeVene <<a href="mailto:David.LeVene@blackboard.com" target="_blank">David.LeVene@blackboard.com</a>><br>
<b>Cc:</b> <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>; Dan Kenigsberg <<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>><br>
<b>Subject:</b> Re: [ovirt-users] Configuring the SRIOV virsh device</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hi again David)
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">In order to achieve such xml ^^ (the one you describing) you should first of all prepare a proper 'Port Profile' on your UCS manager side.
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">- In your/from your example you should have 2 Port Profiles([1], [2]) configured in your cisco side(see -
<a href="http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/kvm/gui/config_guide/2-1/b_GUI_KVM_VM-FEX_UCSM_Configuration_Guide_2_1/b_GUI_KVM_VM-FEX_UCSM_Configuration_Guide_2_1_chapter_010.html" target="_blank">
http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/kvm/gui/config_guide/2-1/b_GUI_KVM_VM-FEX_UCSM_Configuration_Guide_2_1/b_GUI_KVM_VM-FEX_UCSM_Configuration_Guide_2_1_chapter_010.html</a>)
<br>
<br>
[1] profile-ame1-test1<br>
[2] profile-ame1-prep1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- This port profiles should associate with your service profiles(blades)<br>
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- If you have those port profiles on your ucs side and they well configured, then you should follow the steps i suggested in the previous e-mail -->
<br>
<br>
1) Run for example:<br>
engine-config -s CustomDeviceProperties="{type=interface;prop={ifacemacspoof=^(true|false)$;queues=[1-9][0-9]*;vmfex=^[a-zA-Z0-9_.-]{2,32}$;SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}, *)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}}"
--cver=3.5<br>
<br>
- Restart your ovirt-engine service<br>
- with --cver=3.5 you can specify your cluster version level. ^^<br>
- If you don't want that it will overwrite your current attributes, you have to send them all as well(like on my example above). ^^<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">We are using 3.6.3, with the applied patch from
<a href="https://gerrit.ovirt.org/#/c/54237" target="_blank">https://gerrit.ovirt.org/#/c/54237</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"># engine-config -g CustomDeviceProperties</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: version: 3.0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: version: 3.1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: version: 3.2</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: version: 3.3</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: {type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.4</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: {type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.5</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">CustomDeviceProperties: {type=interface;prop={ifacemacspoof=^(true|false)$;queues=[1-9][0-9]*;vmfex=^[a-zA-Z0-9_.-]{2,32}$;SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.6</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
2) Install vdsm-hook-vmfex-dev on your server(if you don't using vdsm version 3.6, the vmfex hook isn't installed by default)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"># rpm -qa | grep -i vmfex</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">vdsm-hook-vmfex-dev-4.17.23-1.el7.noarch</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">vdsm-hook-vmfex-4.17.23-0.el7.centos.noarch</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
3) In ovirt engine you can verify that host have the hooks installed under 'Hosts' main tab - > 'Host Hooks' sub tab<br>
You should see there: a) before_device_migrate_destination b) before_device_create c) before_nic_hotplug<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img border="0" width="896" height="248" id="_x0000_i1025" src="cid:image003.png@01D17B0F.3AE61C60"><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
4) Create new network and edit the vNIC profile. <br>
Choose from the CustomDeviceProperties the 'vmfex' key(like in your example) and in the right field enter your Port Profile id/name(the one that is configured on your UCS side)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img border="0" width="563" height="406" id="_x0000_i1026" src="cid:image005.jpg@01D17B0F.3AE61C60"><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
- 'Networks' main tab(create network) >> 'vNIC Profiles' sub tab(edit the profile and add the 'vmfex' key), so it would look like:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">vmfex = profile-ame1-test1<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">and/or
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">vmfex = profile-ame1-prep1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img border="0" width="473" height="319" id="_x0000_i1027" src="cid:image006.png@01D17B0F.3AE61C60"><o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
5) Go to 'Setup Networks' sub tab under 'Hosts' main tab --> attach the network with the vmfex profile to NIC on host(drag it), it should be the additional NIC, not the one that the management network attached to.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><img border="0" width="888" height="121" id="_x0000_i1028" src="cid:image007.png@01D17B0F.3AE61C60"><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><img border="0" width="613" height="414" id="_x0000_i1029" src="cid:image008.jpg@01D17B0F.3AE61C60"><br>
<br>
6) Create VM or use exist one, create/add new vNIC to the VM with the vmfex profile<span style="color:black"><br>
</span><img border="0" width="635" height="423" id="_x0000_i1030" src="cid:image009.jpg@01D17B0F.3AE61C60"><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<br>
7) Run VM<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Note - only Test hosts 2 & 3 are currently configured with this network - so I expect Host 1 to fail.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><img border="0" width="490" height="372" id="_x0000_i1031" src="cid:image010.png@01D17B0F.3AE61C60"><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">If your port profile configured properly you should see the proper xml, like you described above ^^ -->
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Good luck and kind regards,
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Michael B<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Wed, Mar 9, 2016 at 8:49 AM, David LeVene <<a href="mailto:David.LeVene@blackboard.com" target="_blank">David.LeVene@blackboard.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hey All,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Still trying to work through this VMFEX stuff, and I know what I want the file to look like at the end.. but not sure how to achieve it from the doco written here<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="http://www.ovirt.org/develop/developer-guide/vdsm/hook/vmfex/" target="_blank">http://www.ovirt.org/develop/developer-guide/vdsm/hook/vmfex/</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">and<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="http://www.ovirt.org/develop/release-management/features/network/ucs-integration/" target="_blank">http://www.ovirt.org/develop/release-management/features/network/ucs-integration/</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Currently my device looks like this<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"># virsh -r net-dumpxml vdsm-SRIOV<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><network><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <name>vdsm-SRIOV</name><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <forward mode='bridge'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <bridge name='SRIOV'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"></network><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I want it looking like this, then the networking will be as it should be!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">A port group would be a vNIC Profile from the looks of things…<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <name>vdsm-SRIOV</name><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <forward dev='enp6s0f1' mode='passthrough'>
<span style="font-family:Wingdings">ß</span> defined as a passthrough device, not a bridge<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <interface dev='enp6s0f1'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <interface dev='enp6s0f2'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <interface dev='enp6s0f3'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <interface dev='enp6s0f4'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">< .. list of interfaces available to it which would need to be manually inputted as a hook><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </forward><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <portgroup name='ame1-test1'><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <virtualport type='802.1Qbh'><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <parameters profileid='profile-ame1-test1'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </virtualport><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </portgroup>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <portgroup name='ame1-prep1'><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <virtualport type='802.1Qbh'><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <parameters profileid='profile-ame1-prep1'/><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </virtualport><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </portgroup><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"></network><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Cheers<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">David<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended
recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Michael Burman<br>
RedHat Israel, RHEV-M QE Network Team<br>
<br>
Mobile: 054-5355725<br>
IRC: mburman<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal">This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use
of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Michael Burman<br>
RedHat Israel, RHEV-M QE Network Team<br>
<br>
Mobile: 054-5355725<br>
IRC: mburman<o:p></o:p></p>
</div>
</div>
</div>
</div>
This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information
is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
</body>
</html>