[Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions. Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale Regards Michael
+1 it will also solve the bug of can do action checks running outside transaction. maybe we can also find a nice way to keep db only commands (without vdsm calls) to run within transaction to avoid repeat the code to open the transaction in such commands. by the way, we should solve the same for the end methods. ----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
"it will also solve the bug of can do action checks running outside" - when not executing it from opened transaction ofcourse :-), but now it happens also by mistake. ----- Original Message -----
From: "Liron Aravot" <laravot@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 1:06:52 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
+1 it will also solve the bug of can do action checks running outside transaction.
maybe we can also find a nice way to keep db only commands (without vdsm calls) to run within transaction to avoid repeat the code to open the transaction in such commands.
by the way, we should solve the same for the end methods.
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Liron Aravot" <laravot@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 1:08:07 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
"it will also solve the bug of can do action checks running outside" - when not executing it from opened transaction ofcourse :-), but now it happens also by mistake.
----- Original Message -----
From: "Liron Aravot" <laravot@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 1:06:52 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
+1 it will also solve the bug of can do action checks running outside transaction.
maybe we can also find a nice way to keep db only commands (without vdsm calls) to run within transaction to avoid repeat the code to open the transaction in such commands.
by the way, we should solve the same for the end methods.
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on
engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
In general - I think the idea is good, we need to take into consideration what you mentioned at 2.
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Yair Zaslavsky" <yzaslavs@redhat.com> To: "Liron Aravot" <laravot@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org>, "Michael Kublin" <mkublin@redhat.com> Sent: Tuesday, February 12, 2013 2:22:09 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
----- Original Message -----
From: "Liron Aravot" <laravot@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 1:08:07 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
"it will also solve the bug of can do action checks running outside" - when not executing it from opened transaction ofcourse :-), but now it happens also by mistake.
----- Original Message -----
From: "Liron Aravot" <laravot@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 1:06:52 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
+1 it will also solve the bug of can do action checks running outside transaction.
maybe we can also find a nice way to keep db only commands (without vdsm calls) to run within transaction to avoid repeat the code to open the transaction in such commands.
by the way, we should solve the same for the end methods.
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on
engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
In general - I think the idea is good, we need to take into consideration what you mentioned at 2.
That's why I wrote about this, we should start at some point. Now, we can divide it to areas of functionality and it will be great to get input about situation in such areas. I know about the following areas: 1. VM related flows (run,stop,migrate,hibernate, etc..) 2. Host flows (maintenance, activate, etc...) 3. Storage flows (activate, deactivate, reconstruct) 4. VM and templates flows (create template, remove template, etc...) 5. Network flows (set up network, add network, etc...) 6. Permissions and so one ....
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
need to remember that transaction also gave us 10-min limit for command execution, not saying its good or bad, but disabling the transactions will allow commands to run forever, this 10 min limit actually, saved us few times when commands got stuck in all kind of locks etc..
----- Original Message -----
From: "Omer Frenkel" <ofrenkel@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 4:32:44 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
need to remember that transaction also gave us 10-min limit for command execution, not saying its good or bad, but disabling the transactions will allow commands to run forever, this 10 min limit actually, saved us few times when commands got stuck in all kind of locks etc..
Not exactly, if transaction has expired the JBOSS transaction framework (Arjuna) will not stop a running thread attached to such thread. We can configure it in such way that a running thread will be cancelled, but for now it is not.
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 4:59:19 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
----- Original Message -----
From: "Omer Frenkel" <ofrenkel@redhat.com> To: "Michael Kublin" <mkublin@redhat.com> Cc: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 4:32:44 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
need to remember that transaction also gave us 10-min limit for command execution, not saying its good or bad, but disabling the transactions will allow commands to run forever, this 10 min limit actually, saved us few times when commands got stuck in all kind of locks etc..
Not exactly, if transaction has expired the JBOSS transaction framework (Arjuna) will not stop a running thread attached to such thread. We can configure it in such way that a running thread will be cancelled, but for now it is not.
So current behavior is somewhat "unexpected" - it all determines if the thread will try to use the transaction after it was "reaped off" from him. Right? If this is the case - I think this is bad. Do we have a real requirement for commands to end their execution phase at 10 minutes?
_______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
----- Original Message -----
From: "Michael Kublin" <mkublin@redhat.com> To: "engine-devel" <engine-devel@ovirt.org> Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - "try to keep transactions as short as possible". 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions.
Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled
Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale
Regards Michael _______________________________________________ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
need to remember that transaction also gave us 10-min limit for command execution, not saying its good or bad, but disabling the transactions will allow commands to run forever, this 10 min limit actually, saved us few times when commands got stuck in all kind of locks etc..
Not exactly, if transaction has expired the JBOSS transaction framework (Arjuna) will not stop a running thread attached to such thread. We can configure it in such way that a running thread will be cancelled, but for now it is not.
So current behavior is somewhat "unexpected" - it all determines if the thread will try to use the transaction after it was "reaped off" from him. Right? If this is the case - I think this is bad. Do we have a real requirement for commands to end their execution phase at 10 minutes?
Propose patch for removing global transaction at endAction of CommandBase (this is a first step) http://gerrit.ovirt.org/#/c/12010/
participants (4)
-
Liron Aravot -
Michael Kublin -
Omer Frenkel -
Yair Zaslavsky