Hi fellow oVirters, On this test day I picked JSON RPC for testing: My initial environment consisted of a hosted engine setup with three hosts. Upgrading =========== The first task, then, was to upgrade everything to 3.5. My guide was http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine One issue that I encountered was that after setting the 3.5 pre-release yum repo on the engine VM and doing yum update, when doing engine-setup To make it handle the upgrade, that failed complaining about missing a package called patternfly1. I remembered that on the list Greg Sheremeta mentioned a copr repository for it, so I went ahead, installed it and repeated the engine-setup This time it succeeded, however I feel like patternfly1 should probably be an ovirt-engine-3.5 dependency and it should be in the ovirt 3.5 repository. I also noticed that after upgrading the hosts the amount of free system memory is much lower, while the VMs continue to run fine. I opened: https://bugzilla.redhat.com/1124451 Another thing that happened was that restarting ovirt-ha-agent and ovirt-ha-broker using systemd fails silently and it says that they were killed. I only got it to work again by doing: /lib/systemd/systemd-ovirt-ha-broker restart /lib/systemd/systemd-ovirt-ha-agent restart However, and obviously, doing it like that escapes from the advisable control of systemd. Another bad thing is that after doing all this we get that the current host (which has the engine running displays): --== Host 2 status ==-- Status up-to-date : False Hostname : 10.34.63.180 Host ID : 2 Engine status : unknown stale-data Score : 2000 Local maintenance : False Host timestamp : 1406640293 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1406640293 (Tue Jul 29 15:24:53 2014) host-id=2 score=2000 maintenance=False bridge=True cpu-load=0.0856 engine-health={"health": "good", "vm": "up", "detail": "up"} gateway=True mem-free=3192 Why did the engine status go to unknown stale-data? (it also happened for the other two hosts in the setup. Changing hosts to use JSON RPC =============================== After talking with Piotr and trying it with the webadmin UI, I found out that there is no direct way to update a host's settings to use json rpc as its connectivity mechanism with the engine. This constitutes a usabiltiy bug which I filed: https://bugzilla.redhat.com/1124442 It is presumable that users will want to move to the newer and better RPC mechanism and they should be able to do so by merely putting the host in maintenance and ticking some checkbox in the 'edit host' dialog. I did the workaround of removing the host and adding it again and that worked, the host went up. Network operations via JSON RPC ================================ After the host went up, I decided to send a setupNetworks command. The operation worked out fine, but unfortunately we have a very serious gap that makes it impossible for me to use jsonrpc for network operations/development. **Logging** When doing a network operation with xmlrpc we'd get the following in vdsm.log Thread-21::DEBUG::2014-07-30 13:38:11,414::BindingXMLRPC::1127::vds::(wrapper) client [10.34.61.242]::call setupNetworks with ({'10': {'nic': 'em2', 'vlan': '10', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [686033d4] Thread-21::DEBUG::2014-07-30 13:38:32,689::BindingXMLRPC::1134::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}} As you can see, we get the bare minimum logging one could ask for, an entry with the command called and the data it received and another entry with the return result data. Doing the same with jsonrpc (and ignoring the excessive IOProcess) if I search for "setupNetworks" the only thing I get is: Thread-23057::DEBUG::2014-07-30 13:32:44,126::__init__::462::jsonrpc.JsonRpcServer::(_serveRequest) Looking for method 'Host_setupNetworks' in bridge And if I search for the data received, like 'STP', there is nothing whatsoever. As I said, unless this is fixed and we get entries with the same amount of data as before, it can't be used in production nor in development. https://bugzilla.redhat.com/1124813 TL;DR: - lack of usability upgrading an environment to use jsonrpc https://bugzilla.redhat.com/1124442 - Failure on step 7 of upgrade steps: https://bugzilla.redhat.com/1124826 - JSON RPC logging excessive but insuficient for network call debugging https://bugzilla.redhat.com/1124813
----- Original Message -----
From: "Antoni Segura Puimedon" <asegurap@redhat.com> To: devel@ovirt.org, "users" <users@ovirt.org> Sent: Wednesday, July 30, 2014 2:15:36 PM Subject: [ovirt-users] Testday: json rpc [...] **Logging**
When doing a network operation with xmlrpc we'd get the following in vdsm.log Thread-21::DEBUG::2014-07-30 13:38:11,414::BindingXMLRPC::1127::vds::(wrapper) client [10.34.61.242]::call setupNetworks with ({'10': {'nic': 'em2', 'vlan': '10', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [686033d4] Thread-21::DEBUG::2014-07-30 13:38:32,689::BindingXMLRPC::1134::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}
As you can see, we get the bare minimum logging one could ask for, an entry with the command called and the data it received and another entry with the return result data.
Doing the same with jsonrpc (and ignoring the excessive IOProcess) if I search for "setupNetworks" the only thing I get is: Thread-23057::DEBUG::2014-07-30 13:32:44,126::__init__::462::jsonrpc.JsonRpcServer::(_serveRequest) Looking for method 'Host_setupNetworks' in bridge
And if I search for the data received, like 'STP', there is nothing whatsoever. As I said, unless this is fixed and we get entries with the same amount of data as before, it can't be used in production nor in development.
Can you share some VDSM logs? Attaching them to BZ would be super. I'd like to check if that applies to virt flows as well - as I actually tend to believe. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "Antoni Segura Puimedon" <asegurap@redhat.com> Cc: devel@ovirt.org, "users" <users@ovirt.org> Sent: Wednesday, July 30, 2014 2:59:37 PM Subject: Re: [ovirt-users] Testday: json rpc
----- Original Message -----
From: "Antoni Segura Puimedon" <asegurap@redhat.com> To: devel@ovirt.org, "users" <users@ovirt.org> Sent: Wednesday, July 30, 2014 2:15:36 PM Subject: [ovirt-users] Testday: json rpc [...] **Logging**
When doing a network operation with xmlrpc we'd get the following in vdsm.log Thread-21::DEBUG::2014-07-30 13:38:11,414::BindingXMLRPC::1127::vds::(wrapper) client [10.34.61.242]::call setupNetworks with ({'10': {'nic': 'em2', 'vlan': '10', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [686033d4] Thread-21::DEBUG::2014-07-30 13:38:32,689::BindingXMLRPC::1134::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}
As you can see, we get the bare minimum logging one could ask for, an entry with the command called and the data it received and another entry with the return result data.
Doing the same with jsonrpc (and ignoring the excessive IOProcess) if I search for "setupNetworks" the only thing I get is: Thread-23057::DEBUG::2014-07-30 13:32:44,126::__init__::462::jsonrpc.JsonRpcServer::(_serveRequest) Looking for method 'Host_setupNetworks' in bridge
And if I search for the data received, like 'STP', there is nothing whatsoever. As I said, unless this is fixed and we get entries with the same amount of data as before, it can't be used in production nor in development.
Can you share some VDSM logs? Attaching them to BZ would be super.
Good point, I'll do that now.
I'd like to check if that applies to virt flows as well - as I actually tend to believe.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: "Antoni Segura Puimedon" <asegurap@redhat.com> Cc: "users" <users@ovirt.org>, devel@ovirt.org Sent: Wednesday, July 30, 2014 3:59:37 PM Subject: Re: [ovirt-devel] [ovirt-users] Testday: json rpc
----- Original Message -----
From: "Antoni Segura Puimedon" <asegurap@redhat.com> To: devel@ovirt.org, "users" <users@ovirt.org> Sent: Wednesday, July 30, 2014 2:15:36 PM Subject: [ovirt-users] Testday: json rpc [...] **Logging**
When doing a network operation with xmlrpc we'd get the following in vdsm.log Thread-21::DEBUG::2014-07-30 13:38:11,414::BindingXMLRPC::1127::vds::(wrapper) client [10.34.61.242]::call setupNetworks with ({'10': {'nic': 'em2', 'vlan': '10', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [686033d4] Thread-21::DEBUG::2014-07-30 13:38:32,689::BindingXMLRPC::1134::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}
As you can see, we get the bare minimum logging one could ask for, an entry with the command called and the data it received and another entry with the return result data.
Doing the same with jsonrpc (and ignoring the excessive IOProcess) if I search for "setupNetworks" the only thing I get is: Thread-23057::DEBUG::2014-07-30 13:32:44,126::__init__::462::jsonrpc.JsonRpcServer::(_serveRequest) Looking for method 'Host_setupNetworks' in bridge
And if I search for the data received, like 'STP', there is nothing whatsoever. As I said, unless this is fixed and we get entries with the same amount of data as before, it can't be used in production nor in development.
Can you share some VDSM logs? Attaching them to BZ would be super.
You can look also here for vdsm logs when using jsonrpc: https://bugzilla.redhat.com/attachment.cgi?id=922463 https://bugzilla.redhat.com/attachment.cgi?id=922347
I'd like to check if that applies to virt flows as well - as I actually tend to believe.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Antoni Segura Puimedon" <asegurap@redhat.com> To: devel@ovirt.org, "users" <users@ovirt.org> Sent: Wednesday, 30 July, 2014 2:15:36 PM Subject: Testday: json rpc
Hi fellow oVirters,
On this test day I picked JSON RPC for testing:
My initial environment consisted of a hosted engine setup with three hosts.
Upgrading =========== The first task, then, was to upgrade everything to 3.5.
My guide was http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine
One issue that I encountered was that after setting the 3.5 pre-release yum repo on the engine VM and doing yum update, when doing engine-setup To make it handle the upgrade, that failed complaining about missing a package called patternfly1. I remembered that on the list Greg Sheremeta mentioned a copr repository for it, so I went ahead, installed it and repeated the engine-setup
This time it succeeded, however I feel like patternfly1 should probably be an ovirt-engine-3.5 dependency and it should be in the ovirt 3.5 repository.
I also noticed that after upgrading the hosts the amount of free system memory is much lower, while the VMs continue to run fine. I opened: https://bugzilla.redhat.com/1124451
Another thing that happened was that restarting ovirt-ha-agent and ovirt-ha-broker using systemd fails silently and it says that they were killed. I only got it to work again by doing:
/lib/systemd/systemd-ovirt-ha-broker restart /lib/systemd/systemd-ovirt-ha-agent restart
However, and obviously, doing it like that escapes from the advisable control of systemd. Another bad thing is that after doing all this we get that the current host (which has the engine running displays):
--== Host 2 status ==--
Status up-to-date : False Hostname : 10.34.63.180 Host ID : 2 Engine status : unknown stale-data Score : 2000 Local maintenance : False Host timestamp : 1406640293 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1406640293 (Tue Jul 29 15:24:53 2014) host-id=2 score=2000 maintenance=False bridge=True cpu-load=0.0856 engine-health={"health": "good", "vm": "up", "detail": "up"} gateway=True mem-free=3192
Why did the engine status go to unknown stale-data? (it also happened for the other two hosts in the setup.
After discussing with Jiři: the 'stale data thing' resulted in https://bugzilla.redhat.com/1125244
Changing hosts to use JSON RPC ===============================
After talking with Piotr and trying it with the webadmin UI, I found out that there is no direct way to update a host's settings to use json rpc as its connectivity mechanism with the engine. This constitutes a usabiltiy bug which I filed:
https://bugzilla.redhat.com/1124442
It is presumable that users will want to move to the newer and better RPC mechanism and they should be able to do so by merely putting the host in maintenance and ticking some checkbox in the 'edit host' dialog.
I did the workaround of removing the host and adding it again and that worked, the host went up.
Network operations via JSON RPC ================================
After the host went up, I decided to send a setupNetworks command. The operation worked out fine, but unfortunately we have a very serious gap that makes it impossible for me to use jsonrpc for network operations/development.
**Logging**
When doing a network operation with xmlrpc we'd get the following in vdsm.log Thread-21::DEBUG::2014-07-30 13:38:11,414::BindingXMLRPC::1127::vds::(wrapper) client [10.34.61.242]::call setupNetworks with ({'10': {'nic': 'em2', 'vlan': '10', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [686033d4] Thread-21::DEBUG::2014-07-30 13:38:32,689::BindingXMLRPC::1134::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}
As you can see, we get the bare minimum logging one could ask for, an entry with the command called and the data it received and another entry with the return result data.
Doing the same with jsonrpc (and ignoring the excessive IOProcess) if I search for "setupNetworks" the only thing I get is: Thread-23057::DEBUG::2014-07-30 13:32:44,126::__init__::462::jsonrpc.JsonRpcServer::(_serveRequest) Looking for method 'Host_setupNetworks' in bridge
And if I search for the data received, like 'STP', there is nothing whatsoever. As I said, unless this is fixed and we get entries with the same amount of data as before, it can't be used in production nor in development.
https://bugzilla.redhat.com/1124813
TL;DR: - lack of usability upgrading an environment to use jsonrpc https://bugzilla.redhat.com/1124442 - Failure on step 7 of upgrade steps: https://bugzilla.redhat.com/1124826 - JSON RPC logging excessive but insuficient for network call debugging https://bugzilla.redhat.com/1124813
participants (3)
-
Antoni Segura Puimedon -
Francesco Romani -
Nir Soffer