1:43 a.m.
Hi Michael,
I've made progress, but it's still not 100% working but I'm nearly there. As
this is currently an investigation/side project I haven't had time to progress
further.
I did setup the 2 vNIC's in the UCS and that's working fine, and my setup
isn't far off of what you have shown me in your screen shots (really helpful,
thankyou!)
And I'll post again if I come across any further issues that I get stuck on.
Thanks again for your help so far.
Regards
David
-----Original Message-----
From: Michael Burman [mailto:mburman@redhat.com]
Sent: Monday, February 22, 2016 20:20
To: David LeVene <David.LeVene(a)blackboard.com>
Cc: users(a)ovirt.org; Dan Kenigsberg <danken(a)redhat.com>
Subject: Re: [ovirt-users] UCS Integration (vmfex/SRIOV)
Hi David,
I hope that everything working for you)
From your description it seems like you have only 1 vNIC on your UCS blade and that it is
a problem. You should enable an extra vNIC on your UCS blade(service profile).
I'm not sure i understood what exactly happened on your setup, but i think it is
because you have only 1 vNIC and you trying to use it as the management network and the
vmfex hook.
you should have 1 vNIC for the management network --> eth0=ovirtmgmt and 1 vNIC for the
network with a vmfex profile --> eth1='vmfex network'(network with
vmfex=PortProfile)
Attaching screenshots as an example of how it should look like.
Michael B
----- Original Message -----
From: "David LeVene" <David.LeVene(a)blackboard.com>
To: "Michael Burman" <mburman(a)redhat.com>
Cc: users(a)ovirt.org
Sent: Thursday, February 18, 2016 7:52:25 AM
Subject: RE: [ovirt-users] UCS Integration (vmfex/SRIOV)
So I found some additional logs.. and from what I understand it’s trying to delete the
ovirtmgmt network from libvirt setup.. but there’s no network configs in there..
ConfigNetworkError: (27, "Cannot delete network 'ovirtmgmt': It doesn't
exist in the system")
restore-net::DEBUG::2016-02-18 15:53:01,883::libvirtconnection::160::root::(get) trying to
connect libvirt
restore-net::INFO::2016-02-18
15:53:02,090::vdsm-restore-net-config::86::root::(_restore_sriov_numvfs) SRIOV network
device which is not persisted found at: 0000:07:00.0.
restore-net::INFO::2016-02-18 15:53:02,090::vdsm-restore-net-config::385::root::(restore)
starting network restoration.
restore-net::DEBUG::2016-02-18
15:53:02,090::vdsm-restore-net-config::193::root::(_remove_networks_in_running_config)
Removing all networks ({'ovirtmgmt': {'remove': True}}) and bonds ({}) in
running config.
restore-net::DEBUG::2016-02-18 15:53:02,091::api::886::setupNetworks::(setupNetworks)
Setting up network according to configuration: networks:{'ovirtmgmt':
{'remove': True}}, bondings:{}, options:{'_inRollback': True,
'connectivityCheck': False}
restore-net::DEBUG::2016-02-18 15:53:02,091::api::890::root::(setupNetworks) Validating
configuration
restore-net::DEBUG::2016-02-18 15:53:02,239::api::900::setupNetworks::(setupNetworks)
Applying...
restore-net::INFO::2016-02-18 15:53:02,239::netconfpersistence::73::root::(setBonding)
Adding bond0({'nics': [], 'options': 'mode=0'})
restore-net::ERROR::2016-02-18 15:53:02,240::__init__::58::root::(__exit__) Failed
rollback transaction last known good network.
Traceback (most recent call last):
File "/usr/share/vdsm/network/api.py", line 932, in setupNetworks
"system" % network)
ConfigNetworkError: (27, "Cannot delete network 'ovirtmgmt': It doesn't
exist in the system")
restore-net::ERROR::2016-02-18 15:53:02,240::vdsm-restore-net-config::393::root::(restore)
unified restoration failed.
Traceback (most recent call last):
File "/usr/share/vdsm/vdsm-restore-net-config", line 388, in restore
unified_restoration()
File "/usr/share/vdsm/vdsm-restore-net-config", line 110, in
unified_restoration
_remove_networks_in_running_config()
File "/usr/share/vdsm/vdsm-restore-net-config", line 195, in
_remove_networks_in_running_config
_inRollback=True)
File "/usr/share/vdsm/network/api.py", line 932, in setupNetworks
"system" % network)
ConfigNetworkError: (27, "Cannot delete network 'ovirtmgmt': It doesn't
exist in the system")
So.. I manually edited the /etc/sysconfig/network-scripts and setup the device &
bridge as expected.. but vdsm didn’t like it.. it was still trying to delete the
ovirtmgmt device.
What I ended up doing in the end is defining the network in virsh (creating myself an
account via saslpasswd2 -a libvirt USERNAME) then creating the ovirtmgmt bridge # cat
vdsm-ovirtmgmt.xml <network>
<name>vdsm-ovirtmgmt</name>
<forward mode='bridge'/>
<bridge name='ovirtmgmt'/>
</network>
#virsh net-define vdsm-ovirtmgmt.xml
#virsh net-start vdsm-ovirtmgmt
Something still wasn’t right here.. so I ended up rebooting the host… and then it was
working.. I’m still not entirely sure what I did to fix it.. but it’s working now…
I’ll continue on the journey and keep this thread updated incase anyone else runs into the
same issues!
Cheers
David
From: David LeVene
Sent: Thursday, February 18, 2016 15:22
To: 'Michael Burman' <mburman(a)redhat.com>
Cc: users(a)ovirt.org
Subject: RE: [ovirt-users] UCS Integration (vmfex/SRIOV)
Hi Michael,
Thanks for this information - it's great, the guide should be updated on the ovirt
webpage to reflect the information you've provided below.
Just to hopefully help someone else out in the future - we had only 1 vNIC defined in the
UCS Service Profile, this caused problems when we enabled dynamic vnic's on that
host.
The resolution was leave the initial interface as it is.. and as a management interface
and setup another interface for the dynamic vnics. Ensure that your "vNIC/vHBA
Placement" doesn't change either!
[cid:image001.png@01D16A64.FCDB9D60]
So in this case eth0 = ovirtmgmt bridge, and ovirt device is the vNICs.
When the system reboots after this change I’ve lost my bridge and vdsmd failed to start.
I manually setup the first interface again so I can talk to it again and I’ve even setup
the bridge.... but I can’t get it working again? Below is what I see in the logs. I’ve
even tried a re-installation(ovirt maint/reinstall) but no joy..
I’ve managed to get this working on 1 host.. but no idea what I did that was different!?..
If I remove the vNIC ovirt it starts working as expected - where can I find out some more
information on what vdsm is doing?
systemd: Started Auxiliary vdsm service for running helper functions as root.
systemd: Starting Auxiliary vdsm service for running helper functions as root...
systemd: Starting Virtualization daemon...
journal: libvirt version: 1.2.17, package: 13.el7_2.2 (CentOS BuildSystem
<http://bugs.centos.org>;, 2015-12-09-13:08:22, worker1.bsys.centos.org)
journal: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not
accessible
systemd: Started Virtualization daemon.
systemd: Starting Virtual Desktop Server Manager network restoration...
vdsm-tool: Traceback (most recent call last):
vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 425, in
<module>
vdsm-tool: restore(args)
vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 388, in
restore
vdsm-tool: unified_restoration()
vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 110, in
unified_restoration
vdsm-tool: _remove_networks_in_running_config()
vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 195, in
_remove_networks_in_running_config
vdsm-tool: _inRollback=True)
vdsm-tool: File "/usr/share/vdsm/network/api.py", line 932, in setupNetworks
vdsm-tool: "system" % network)
vdsm-tool: network.errors.ConfigNetworkError: (27, "Cannot delete network
'ovirtmgmt': It doesn't exist in the system")
vdsm-tool: Traceback (most recent call last):
vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main
vdsm-tool: return tool_command[cmd]["command"](*args)
vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py",
line 41, in restore_command
vdsm-tool: exec_restore(cmd)
vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py",
line 54, in exec_restore
vdsm-tool: raise EnvironmentError('Failed to restore the persisted networks')
vdsm-tool: EnvironmentError: Failed to restore the persisted networks
systemd: vdsm-network.service: main process exited, code=exited, status=1/FAILURE
systemd: Failed to start Virtual Desktop Server Manager network restoration.
systemd: Dependency failed for Virtual Desktop Server Manager.
systemd: Dependency failed for MOM instance configured for VDSM purposes.
systemd: Job mom-vdsm.service/start failed with result 'dependency'.
systemd: Job vdsmd.service/start failed with result 'dependency'.
systemd: Unit vdsm-network.service entered failed state.
systemd: vdsm-network.service failed.
systemd-logind: Removed session 5.
Cheers
David
-----Original Message-----
From: Michael Burman [mailto:mburman@redhat.com]
Sent: Wednesday, February 17, 2016 16:56
To: David LeVene
<David.LeVene@blackboard.com<mailto:David.LeVene@blackboard.com>>
Cc: users@ovirt.org<mailto:users@ovirt.org>
Subject: Re: [ovirt-users] UCS Integration (vmfex/SRIOV)
HI David)
If everything configured properly on your UCS side and you have a working Port Profile
id/name your next steps should be:
1) Run for example:
engine-config -s
CustomDeviceProperties="{type=interface;prop={ifacemacspoof=^(true|false)$;queues=[1-9][0-9]*;vmfex=^[a-zA-Z0-9_.-]{2,32}$;SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}}" --cver=3.5
- Restart your ovirt-engine service
- with --cver=3.5 you can specify your cluster version level. ^^
- If you don't want that it will overwrite your current attributes, you have to send
them all as well(like on my example above). ^^
2) Install vdsm-hook-vmfex-dev on your nodes
3) In ovirt engine you can verify that host have the hooks installed under
'Hosts' main tab - > 'Host Hooks' sub tab You should see there: a)
before_device_migrate_destination b) before_device_create c) before_nic_hotplug
4) Create new network and edit the vNIC profile. Choose from the CustomDeviceProperties
the 'vmfex' key(like in your example) and in the right field enter your Port
Profile id/name(the one that is configured on your UCS side)
5) Go to 'Setup Networks' sub tab under 'Hosts' -> attach the network
with the vmfex profile to NIC on host(drag it)
6) Create VM or use exist one, create/add new vNIC to the VM with the vmfex profile
7) Run VM
Good luck)
----- Original Message -----
From: "David LeVene"
<David.LeVene@blackboard.com<mailto:David.LeVene@blackboard.com>>
To: users@ovirt.org<mailto:users@ovirt.org>
Sent: Wednesday, February 17, 2016 5:41:21 AM
Subject: [ovirt-users] UCS Integration (vmfex/SRIOV)
Hey All,
I’ve got a test instance and would like to get vmfex working. I’m very new to ovirt, so
some of the basics are still new to me.
I currently have a 3 node setup which are all configured without vmfex, what I’d like to
do is convert the hosts one by one to use vmfex.
From the UCS side of things it works and I’m presented with the SRIOV devices in the
host.. but I can’t get the network config in the ovirt admin setup..
I’m following this guide, but not getting very far…
http://www.ovirt.org/Features/UCS_Integration
This step says to do this, which is fine.. but when I do this I overwrite the current
attribute in there.
engine-config -s
CustomDeviceProperties='{type=interface;prop={vmfex=^[a-zA-Z0-9_.-]{2,32}$}}'
which is…
# engine-config -g CustomDeviceProperties
CustomDeviceProperties: version: 3.0
CustomDeviceProperties: version: 3.1
CustomDeviceProperties: version: 3.2
CustomDeviceProperties: version: 3.3
CustomDeviceProperties:
{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.4
CustomDeviceProperties:
{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.5
CustomDeviceProperties:
{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.6
When I update it using the command above, it replaces it…
When I append the 2 together, I get see vmfex & prop in the ovirt-engine (screen shot)
side of things.. so can someone please check my command to ensure I’m not missing
something!
engine-config -s
CustomDeviceProperties='{type=interface;prop={vmfex=^[a-zA-Z0-9_.-]{2,32}$}}{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}}’
# engine-config -g CustomDeviceProperties
CustomDeviceProperties: version: 3.0
CustomDeviceProperties: version: 3.1
CustomDeviceProperties: version: 3.2
CustomDeviceProperties: version: 3.3
CustomDeviceProperties:
{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.4
CustomDeviceProperties:
{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.5
CustomDeviceProperties:
{type=interface;prop={vmfex=^[a-zA-Z0-9_.-]{2,32}$}}{type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12},
*)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$}} version: 3.6
As soon as I enable vmfex on a host that was previously working.. it stops working and the
network no longer works on the host… (this is not using any of the config above.. just
enabled the vmfex in the UCS manager)
What is the “correct” way to bring a Host in ovirt using vmfex?
Regards
David
This email and any attachments may contain confidential and proprietary information of
Blackboard that is for the sole use of the intended recipient. If you are not the intended
recipient, disclosure, copying, re-distribution or other use of any of this information is
strictly prohibited. Please immediately notify the sender and delete this transmission if
you received this email in error.
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--
Michael Burman
RedHat Israel, RHEV-M QE Network Team
Mobile: 054-5355725
IRC: mburman
This email and any attachments may contain confidential and proprietary information of
Blackboard that is for the sole use of the intended recipient. If you are not the intended
recipient, disclosure, copying, re-distribution or other use of any of this information is
strictly prohibited. Please immediately notify the sender and delete this transmission if
you received this email in error.
--
Michael Burman
RedHat Israel, RHEV-M QE Network Team
Mobile: 054-5355725
IRC: mburman
This email and any attachments may contain confidential and proprietary information of
Blackboard that is for the sole use of the intended recipient. If you are not the intended
recipient, disclosure, copying, re-distribution or other use of any of this information is
strictly prohibited. Please immediately notify the sender and delete this transmission if
you received this email in error.