OVIRT-3.5-TEST-DAY-3: replace XML-rpc with JSON-rpc
Hi everyone of this test day I played with the new JSON-rpc transport, which is enabled by default. I set up from scratch two hypervisors running on RHEL 6.5, while Engine was running on F19. I created everything from scratch and tested all the flows I known good enough, including less-then-common stuff. The list includes creating storage domains, VMs, hotplug/unplug disks, snapshots, migration, hibernation and dehibernation. On top of that, I prepared an environment to run more tests, that included create 200 VMs, started/stopped 100 of them to test. I kept an eye on the logs other than Engine. Everything I tried went OK, and logs look good to me. I run in a few hiccups, which I mention for the sake of completeness: - VDSM refused to start or run VMs initially: libvirt config included relics from past environment on the same box, not JSON-rpc fault. Fixed with new config and (later) a reboot. - Trying recovery, Engine took longer than expected to sync up with VDSM. I have not hard data and feeling is not enough to file a BZ, so I didn't. - Still trying recovery, one and just one time Engine had stale data from VDSM (reported two VMs as present which actually aren't). Not sure it was related to JSON-rpc, can't reproduce, so not filed a BZ. Since I need to continue profiling anyway (better late than never I guess), I include a couple of test to see if JSON-rpc makes any difference to JSON-rpc. Results will be posted on devel@ovirt.org and/or on the oVirt wiki. -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Cc: "users" <users@ovirt.org> Sent: Wednesday, September 17, 2014 5:33:01 PM Subject: [ovirt-users] OVIRT-3.5-TEST-DAY-3: replace XML-rpc with JSON-rpc
Everything I tried went OK, and logs look good to me.
I run in a few hiccups, which I mention for the sake of completeness: - VDSM refused to start or run VMs initially: libvirt config included relics from past environment on the same box, not JSON-rpc fault. Fixed with new config and (later) a reboot. - Trying recovery, Engine took longer than expected to sync up with VDSM. I have not hard data and feeling is not enough to file a BZ, so I didn't. - Still trying recovery, one and just one time Engine had stale data from VDSM (reported two VMs as present which actually aren't). Not sure it was related to JSON-rpc, can't reproduce, so not filed a BZ.
I need to partially amend this statement as, running more benchmarks/profiling, I got this twice in a row INFO:root:starting 100 vms INFO:root:start: serial execution INFO:root:Starting VM: XS_C000 INFO:root:Starting VM: XS_C001 INFO:root:Starting VM: XS_C002 Traceback (most recent call last): File "./observe.py", line 154, in <module> data = bench(host, 'XS_C%03i', first, last, api, outfile, mins * 60.) File "./observe.py", line 122, in bench start(vms) File "./observe.py", line 66, in start vm.start() File "./observe.py", line 54, in start self._handle.start() File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16507, in start headers={"Correlation-Id":correlation_id} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 118, in request persistent_auth=self._persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 140, in __doRequest persistent_auth=persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/web/connection.py", line 134, in doRequest raise RequestError, response ovirtsdk.infrastructure.errors.RequestError: status: 400 reason: Bad Request detail: Network error during communication with the Host. (this is a runner script using ovirt sdk for python, source is available on demand and will be published anyway soon[ish]) On engine logs I see something alike this: http://fpaste.org/134263/ Since the above is way too vague to file a meaningful BZ, I'm now continuing the investigation to see if there is a bug somewhere or if it's a hiccup of my local environment. Thanks, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
On 17/09/14 15:46 -0400, Francesco Romani wrote:
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Cc: "users" <users@ovirt.org> Sent: Wednesday, September 17, 2014 5:33:01 PM Subject: [ovirt-users] OVIRT-3.5-TEST-DAY-3: replace XML-rpc with JSON-rpc
Everything I tried went OK, and logs look good to me.
I run in a few hiccups, which I mention for the sake of completeness: - VDSM refused to start or run VMs initially: libvirt config included relics from past environment on the same box, not JSON-rpc fault. Fixed with new config and (later) a reboot. - Trying recovery, Engine took longer than expected to sync up with VDSM. I have not hard data and feeling is not enough to file a BZ, so I didn't. - Still trying recovery, one and just one time Engine had stale data from VDSM (reported two VMs as present which actually aren't). Not sure it was related to JSON-rpc, can't reproduce, so not filed a BZ.
I need to partially amend this statement as, running more benchmarks/profiling, I got this twice in a row
INFO:root:starting 100 vms INFO:root:start: serial execution INFO:root:Starting VM: XS_C000 INFO:root:Starting VM: XS_C001 INFO:root:Starting VM: XS_C002
Traceback (most recent call last): File "./observe.py", line 154, in <module> data = bench(host, 'XS_C%03i', first, last, api, outfile, mins * 60.) File "./observe.py", line 122, in bench start(vms) File "./observe.py", line 66, in start vm.start() File "./observe.py", line 54, in start self._handle.start() File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 16507, in start headers={"Correlation-Id":correlation_id} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 118, in request persistent_auth=self._persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 140, in __doRequest persistent_auth=persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/web/connection.py", line 134, in doRequest raise RequestError, response ovirtsdk.infrastructure.errors.RequestError: status: 400 reason: Bad Request detail: Network error during communication with the Host.
(this is a runner script using ovirt sdk for python, source is available on demand and will be published anyway soon[ish])
On engine logs I see something alike this: http://fpaste.org/134263/
Since the above is way too vague to file a meaningful BZ, I'm now continuing the investigation to see if there is a bug somewhere or if it's a hiccup of my local environment.
I just want to note that I have been experiencing vague, intermittent jsonRPC issues with my environment also. I have filed 1143042 which I believe to be a symptom of unreliable communication. It seems to me that we have a definite problem to work out. -- Adam Litke
----- Original Message -----
From: "Adam Litke" <alitke@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, September 17, 2014 10:32:11 PM Subject: Re: [ovirt-devel] [ovirt-users] OVIRT-3.5-TEST-DAY-3: replace XML-rpc with JSON-rpc
On engine logs I see something alike this: http://fpaste.org/134263/
Since the above is way too vague to file a meaningful BZ, I'm now continuing the investigation to see if there is a bug somewhere or if it's a hiccup of my local environment.
I just want to note that I have been experiencing vague, intermittent jsonRPC issues with my environment also. I have filed 1143042 which I believe to be a symptom of unreliable communication. It seems to me that we have a definite problem to work out.
Thanks, subscribed. What is puzzling me is that on my environment everything seemd OK initially (hence the initial report), even the mass start went OK, once. Will update the BZ with my findings, open a new one otherwise. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
On Thu, Sep 18, 2014 at 12:10 AM, Francesco Romani <fromani@redhat.com> wrote:
----- Original Message -----
From: "Adam Litke" <alitke@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: devel@ovirt.org Sent: Wednesday, September 17, 2014 10:32:11 PM Subject: Re: [ovirt-devel] [ovirt-users] OVIRT-3.5-TEST-DAY-3: replace XML-rpc with JSON-rpc
On engine logs I see something alike this: http://fpaste.org/134263/
Since the above is way too vague to file a meaningful BZ, I'm now continuing the investigation to see if there is a bug somewhere or if it's a hiccup of my local environment.
I just want to note that I have been experiencing vague, intermittent jsonRPC issues with my environment also. I have filed 1143042 which I believe to be a symptom of unreliable communication. It seems to me that we have a definite problem to work out.
I checked your log and except of initial communication issues there are no exceptions related to communication. I only found http://ur1.ca/i7g76 and adding Eli to take a look.
Thanks, subscribed.
What is puzzling me is that on my environment everything seemd OK initially (hence the initial report), even the mass start went OK, once.
Will update the BZ with my findings, open a new one otherwise.
Bests,
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: devel@ovirt.org Sent: Wednesday, September 17, 2014 9:46:38 PM Subject: Re: [ovirt-devel] [ovirt-users] OVIRT-3.5-TEST-DAY-3: replace XML-rpc with JSON-rpc
Since the above is way too vague to file a meaningful BZ, I'm now continuing the investigation to see if there is a bug somewhere or if it's a hiccup of my local environment.
I still believe the issue is not clear enough, but anyway I filed https://bugzilla.redhat.com/show_bug.cgi?id=1143968 anyway to move the discussion to a more suitable place and to collect all the evidence. -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
participants (3)
-
Adam Litke -
Francesco Romani -
Piotr Kliczewski