1:09 p.m.
Hi,
we discussed the right amount of logging with Nir and Francesco while reviewing my
patches. Francesco was against one DEBUG log message that could potentially flood the
logs. But the message in question was important for me and SLA, because it was logging the
VDSM changes in response to MoM command.
Since DEBUG really is meant to be used for debug messages, I have one proposal:
1) Change the default log level of vdsm.log to INFO
2) Log (only) DEBUG messages to a separate file vdsm-debug.log
3) Make the vdsm-debug.log rotate faster (every hour, keep only last couple of hours?) so
it does not grow too much
This way the customer would be able to monitor INFO logs (much smaller) without all the
noise and we would be able to collect the DEBUG part in case something happens.
What do you think?
--
Martin Sivák
msivak(a)redhat.com
Red Hat Czech
RHEV-M SLA / Brno, CZ
1:49 p.m.
----- Original Message -----
From: "Martin Sivak" <msivak(a)redhat.com>
To: devel(a)ovirt.org
Sent: Thursday, July 10, 2014 12:09:14 PM
Subject: [ovirt-devel] [vdsm] logging levels and noise in the log
Hi,
we discussed the right amount of logging with Nir and Francesco while
reviewing my patches. Francesco was against one DEBUG log message that could
potentially flood the logs. But the message in question was important for me
and SLA, because it was logging the VDSM changes in response to MoM command.
Since DEBUG really is meant to be used for debug messages, I have one
proposal:
1) Change the default log level of vdsm.log to INFO
2) Log (only) DEBUG messages to a separate file vdsm-debug.log
3) Make the vdsm-debug.log rotate faster (every hour, keep only last couple
of hours?) so it does not grow too much
This way the customer would be able to monitor INFO logs (much smaller)
without all the noise and we would be able to collect the DEBUG part in case
something happens.
What do you think?
+1, because it is an improvement with respect to our current situation.
but we need to carefully pick the debug log rotation frequency.
I also think that we probably need (myself first on the line) a bit more
thoughtful about what we log and on which level.
I believe that log.debug() is used a bit too leniently.
Again, I am the first in the line of guilty.
Bests,
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
1:51 p.m.
Am 10.07.2014 12:09, schrieb Martin Sivak:
Hi,
we discussed the right amount of logging with Nir and Francesco while reviewing my
patches. Francesco was against one DEBUG log message that could potentially flood the
logs. But the message in question was important for me and SLA, because it was logging the
VDSM changes in response to MoM command.
Since DEBUG really is meant to be used for debug messages, I have one proposal:
1) Change the default log level of vdsm.log to INFO
2) Log (only) DEBUG messages to a separate file vdsm-debug.log
3) Make the vdsm-debug.log rotate faster (every hour, keep only last couple of hours?) so
it does not grow too much
This way the customer would be able to monitor INFO logs (much smaller) without all the
noise and we would be able to collect the DEBUG part in case something happens.
What do you think?
From a user perspective I can just encourage you to do so!
Log spam and finding relevant log messages are one of the hugest
drawbacks when using ovirt.
Since now all replies regarding this topic ended like this:
"it is not recommended to lower the default log levels because you
might need this information to debug problems".
But I beg to differ:
If this is a production ready software (after all it is sold somewhere)
you really should not need a log level of "debug" in a productive
environment.
This flag should be for development and test setups, and major breakage
should be fixed there (either in the setup, or in the code) and when
shipped, everyone should be comfortable with a loglevel of info or
warning.
My conclusion was until now: the ovirt devs don't trust their own
implementations to be robust enough and thus require debug flags.
I wonder when all those lurking bugs get fixed that prevent
the log level to get changed to "info" or "warning".
I have the feeling that the devs always want a "debug"
log level, no matter how much the product has matured.
Please correct me where I'm wrong, or share some
insight why these log levels are used and when it will
be possible to lower log levels.
running ovirt, from a log perspective, is like being forced
to run "ssh" always with option "-vvvvvv".
It makes debugging sometimes really hard, because the log
is spammed with so many messages which endusers(admins)
do not even understand that you don't find what you're actually
looking for and you need to waste time to filter the logs to
output what you searched for.
I'm sorry for this long rant, but I really don't understand
the current reasoning for this mess, as a user.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
5:15 p.m.
----- Original Message -----
From: "Martin Sivak" <msivak(a)redhat.com>
To: devel(a)ovirt.org
Sent: Thursday, July 10, 2014 1:09:14 PM
Subject: [ovirt-devel] [vdsm] logging levels and noise in the log
Hi,
we discussed the right amount of logging with Nir and Francesco while
reviewing my patches. Francesco was against one DEBUG log message that could
potentially flood the logs. But the message in question was important for me
and SLA, because it was logging the VDSM changes in response to MoM command.
Since DEBUG really is meant to be used for debug messages, I have one
proposal:
1) Change the default log level of vdsm.log to INFO
Not good enough - our log are full or tracebacks logged in error
and warning levels, which are not helpful to the administrator.
So this will create log that is not useful for anything.
2) Log (only) DEBUG messages to a separate file vdsm-debug.log
The debug log should include *all* log levels, basically being the same as
the current vdsm log. When you need the logs, you want all the information in
the same file.
3) Make the vdsm-debug.log rotate faster (every hour, keep only last
couple
of hours?) so it does not grow too much
I just debugged a problem in production, where the mailbox stopped working
86 logs ago - so having only couple of hours is too little.
However when using proper logging, This issue should be clear even
in info log level, which show error, warnings, and info message such
as mailbox stopping.
This way the customer would be able to monitor INFO logs (much smaller)
without all the noise and we would be able to collect the DEBUG part in case
something happens.
I think you got it right with the dual log approach. We should have two logs:
1. Admin log (events.log?)
This log should show important events and errors, but it should not contain
any tracebacks, thread name, file names etc. Basically, something like the
events log in the engine.
If we take storage for example, this log can contain storage domain state
change, or configuration changes triggered by the engine, or events like
disk that becoming too full.
2. Developer log (vdsm.log)
There is lot of work to to here:
- Disable library logs - we have lot of infrastructure code that log useless
messages. This code should be verified by tests, not by looking at the logs.
- Removing junk log messages:
- new ioprocess logs: https://bugzilla.redhat.com/1114908
- new jsonrpc logs: same issue - no bug yet
- Adding useful info like VDSM changes in response to MoM command.
I did not look at the patch, but if vdsm changes state, it must be logged
- Improving log format
http://gerrit.ovirt.org/28869
- Removing duplicate errors/tracebacks for same error, caused by code that does
try:
something()
except:
log.error('something failed...')
raise
Should be:
something()
And let the upper level handle the failure, logging a trackback.
Or:
try:
something()
Except LowLevelError:
raise FlowError("description...")
FlowError can log a traceback of both errors by using sys.exec_info.
See http://hg.moinmo.in/moin/1.9/file/274a7f675261/MoinMoin/error.py
- Disabling this log by default?
Only if an admin can easily enable or disable the log, ideally from the ui.
So for example, when flow fails, the event log will contain the a single error message
related to the flow, while the debug log will contains the same error message with
a traceback and useful debugging messages during the flow.
Lets make this a feature for 3.6?
Nir
7:07 p.m.
On Thu, Jul 10, 2014 at 10:15:23AM -0400, Nir Soffer wrote:
----- Original Message -----
> From: "Martin Sivak" <msivak(a)redhat.com>
> To: devel(a)ovirt.org
> Sent: Thursday, July 10, 2014 1:09:14 PM
> Subject: [ovirt-devel] [vdsm] logging levels and noise in the log
>
> Hi,
>
> we discussed the right amount of logging with Nir and Francesco while
> reviewing my patches. Francesco was against one DEBUG log message that could
> potentially flood the logs. But the message in question was important for me
> and SLA, because it was logging the VDSM changes in response to MoM command.
>
> Since DEBUG really is meant to be used for debug messages, I have one
> proposal:
>
> 1) Change the default log level of vdsm.log to INFO
Not good enough - our log are full or tracebacks logged in error
and warning levels, which are not helpful to the administrator.
So this will create log that is not useful for anything.
> 2) Log (only) DEBUG messages to a separate file vdsm-debug.log
The debug log should include *all* log levels, basically being the same as
the current vdsm log. When you need the logs, you want all the information in
the same file.
> 3) Make the vdsm-debug.log rotate faster (every hour, keep only last couple
> of hours?) so it does not grow too much
I just debugged a problem in production, where the mailbox stopped working
86 logs ago - so having only couple of hours is too little.
However when using proper logging, This issue should be clear even
in info log level, which show error, warnings, and info message such
as mailbox stopping.
>
> This way the customer would be able to monitor INFO logs (much smaller)
> without all the noise and we would be able to collect the DEBUG part in case
> something happens.
I think you got it right with the dual log approach. We should have two logs:
1. Admin log (events.log?)
This log should show important events and errors, but it should not contain
any tracebacks, thread name, file names etc. Basically, something like the
events log in the engine.
If we take storage for example, this log can contain storage domain state
change, or configuration changes triggered by the engine, or events like
disk that becoming too full.
I have recently introduced a connectivity.log, that tracks changes in
host network devices. Adding parallel logs for other important
admin-understandable changes makes sense.
I'd be happy to hear which bits of information do oVirt admins wish to
see in this "events.log".
2. Developer log (vdsm.log)
There is lot of work to to here:
- Disable library logs - we have lot of infrastructure code that log useless
messages. This code should be verified by tests, not by looking at the logs.
- Removing junk log messages:
- new ioprocess logs: https://bugzilla.redhat.com/1114908
- new jsonrpc logs: same issue - no bug yet
- Adding useful info like VDSM changes in response to MoM command.
I did not look at the patch, but if vdsm changes state, it must be logged
- Improving log format
http://gerrit.ovirt.org/28869
The only problem is to agree which format is better. I personally like
condensed format, that is possibly machine-readable. Spaces, "in
function" and repeating module name and its full path, are suboptimal
imo.
- Removing duplicate errors/tracebacks for same error, caused by code that does
try:
something()
except:
log.error('something failed...')
raise
+1, I hate those.
Should be:
something()
And let the upper level handle the failure, logging a trackback.
Or:
try:
something()
Except LowLevelError:
raise FlowError("description...")
FlowError can log a traceback of both errors by using sys.exec_info.
See http://hg.moinmo.in/moin/1.9/file/274a7f675261/MoinMoin/error.py
- Disabling this log by default?
Only if an admin can easily enable or disable the log, ideally from the ui.
It all too often that a user comes up with a bug or a question. Seeing
his logs usually gives a good hint regarding the solution, or at least
who is to blame. It is important to me to keep this tool handy.
So for example, when flow fails, the event log will contain the a single error message
related to the flow, while the debug log will contains the same error message with
a traceback and useful debugging messages during the flow.
Lets make this a feature for 3.6?
Vdsm has a setLogLevel; it must have rotton out of disuse. But I'd love
to see an Engine option for "cluster debug level", that is overridable
per-host. Each admin should decide what does they care more about:
debugablility by developers, or the bliss of quiet logs.
Dan.
10:13 a.m.
On Jul 10, 2014, at 18:07 , Dan Kenigsberg <danken(a)redhat.com> wrote:
On Thu, Jul 10, 2014 at 10:15:23AM -0400, Nir Soffer wrote:
> ----- Original Message -----
>> From: "Martin Sivak" <msivak(a)redhat.com>
>> To: devel(a)ovirt.org
>> Sent: Thursday, July 10, 2014 1:09:14 PM
>> Subject: [ovirt-devel] [vdsm] logging levels and noise in the log
>>
>> Hi,
>>
>> we discussed the right amount of logging with Nir and Francesco while
>> reviewing my patches. Francesco was against one DEBUG log message that could
>> potentially flood the logs. But the message in question was important for me
>> and SLA, because it was logging the VDSM changes in response to MoM command.
well, that's an easy decision…if it can flood, you have to fix it so it
doesn't;-)
also likely the DEBUG level is not right if you deem it's important
>>
>> Since DEBUG really is meant to be used for debug messages, I have one
>> proposal:
>>
>> 1) Change the default log level of vdsm.log to INFO
>
> Not good enough - our log are full or tracebacks logged in error
> and warning levels, which are not helpful to the administrator.
>
> So this will create log that is not useful for anything.
>
>> 2) Log (only) DEBUG messages to a separate file vdsm-debug.log
>
> The debug log should include *all* log levels, basically being the same as
> the current vdsm log. When you need the logs, you want all the information in
> the same file.
>
>> 3) Make the vdsm-debug.log rotate faster (every hour, keep only last couple
>> of hours?) so it does not grow too much
>
> I just debugged a problem in production, where the mailbox stopped working
> 86 logs ago - so having only couple of hours is too little.
exactly. Let's also not forget the omnipresent ENOSPACE on node…due to logs
If it's debug, it should be likely triggered only while troubleshooting. We should
have a run-time setting for debug levels
>
> However when using proper logging, This issue should be clear even
> in info log level, which show error, warnings, and info message such
> as mailbox stopping.
>
>>
>> This way the customer would be able to monitor INFO logs (much smaller)
>> without all the noise and we would be able to collect the DEBUG part in case
>> something happens.
>
> I think you got it right with the dual log approach. We should have two logs:
>
> 1. Admin log (events.log?)
>
> This log should show important events and errors, but it should not contain
> any tracebacks, thread name, file names etc. Basically, something like the
> events log in the engine.
>
> If we take storage for example, this log can contain storage domain state
> change, or configuration changes triggered by the engine, or events like
> disk that becoming too full.
I have recently introduced a connectivity.log, that tracks changes in
host network devices. Adding parallel logs for other important
admin-understandable changes makes sense.
I wonder if this is the right approach…IMHO it's not, as you have to now take care of
the additional file in the context of your code. It's easy to diverge then.
Why not follow the syslog-like way with config, splitting/limiting files based on e.g.
subpackage prefix, etc…
I'm sure it was brought up many times before….why don't we just send it to
syslog?:)
I'd be happy to hear which bits of information do oVirt admins wish to
see in this "events.log".
>
> 2. Developer log (vdsm.log)
>
> There is lot of work to to here:
>
> - Disable library logs - we have lot of infrastructure code that log useless
> messages. This code should be verified by tests, not by looking at the logs.
>
> - Removing junk log messages:
> - new ioprocess logs: https://bugzilla.redhat.com/1114908
> - new jsonrpc logs: same issue - no bug yet
>
> - Adding useful info like VDSM changes in response to MoM command.
> I did not look at the patch, but if vdsm changes state, it must be logged
>
> - Improving log format
> http://gerrit.ovirt.org/28869
The only problem is to agree which format is better. I personally like
condensed format, that is possibly machine-readable. Spaces, "in
function" and repeating module name and its full path, are suboptimal
imo.
+1
although that was also brought up before IIRC….journald?:)
>
> - Removing duplicate errors/tracebacks for same error, caused by code that does
>
> try:
> something()
> except:
> log.error('something failed...')
> raise
+1, I hate those.
>
> Should be:
>
> something()
>
> And let the upper level handle the failure, logging a trackback.
>
> Or:
>
> try:
> something()
> Except LowLevelError:
> raise FlowError("description...")
>
> FlowError can log a traceback of both errors by using sys.exec_info.
> See http://hg.moinmo.in/moin/1.9/file/274a7f675261/MoinMoin/error.py
>
> - Disabling this log by default?
> Only if an admin can easily enable or disable the log, ideally from the ui.
It all too often that a user comes up with a bug or a question. Seeing
his logs usually gives a good hint regarding the solution, or at least
who is to blame. It is important to me to keep this tool handy.
>
> So for example, when flow fails, the event log will contain the a single error
message
> related to the flow, while the debug log will contains the same error message with
> a traceback and useful debugging messages during the flow.
>
> Lets make this a feature for 3.6?
Vdsm has a setLogLevel; it must have rotton out of disuse. But I'd love
to see an Engine option for "cluster debug level", that is overridable
per-host. Each admin should decide what does they care more about:
debugablility by developers, or the bliss of quiet logs.
per host sounds good for vdsm (btw engine suffers from the same)
I wouldn't bother with UI though….I kind of never understood the tools which have
"enable debug" checkbox in UI but then you anyway have no idea what is it now
creating, what's the impact on the system, where to look for it.
I think debug should remain a purely troubleshooting measure, hence requiring you to log
to host and do something, look/analyze/copy the big dump of info…and then turn it back.
High debug levels typically bring some caveats like less performance…it shouldn't be
something I can blindly check and forgot about it in UI.
Thanks,
michal
Dan.
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
10:24 a.m.
Am 11.07.2014 09:13, schrieb Michal Skrivanek:
I think debug should remain a purely troubleshooting measure, hence
requiring you to log to host and do something, look/analyze/copy the big dump of info…and
then turn it back.
High debug levels typically bring some caveats like less performance…it shouldn't be
something I can blindly check and forgot about it in UI.
You are so right, but yet "debug" is the default shipped log level.
But it's great to see some improvement and discussion around this topic.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
11:03 a.m.
On Jul 11, 2014, at 09:24 , Sven Kieske <S.Kieske(a)mittwald.de> wrote:
Am 11.07.2014 09:13, schrieb Michal Skrivanek:
> I think debug should remain a purely troubleshooting measure, hence requiring you to
log to host and do something, look/analyze/copy the big dump of info…and then turn it
back.
> High debug levels typically bring some caveats like less performance…it shouldn't
be something I can blindly check and forgot about it in UI.
You are so right, but yet "debug" is the default shipped log level.
The big issue right now is that there's no magic solution like "just disable it
and make it warn by default". We really do have to revisit what do we actually want
to show and change the level accordingly. Right now it's too much of a mess, important
stuff is "debug" (as Martin originally hinted even new things are like that:-),
whereas error/warns can flood way too much. I think what Dan did in networking with the
separate file is a good improvement…as it make them/us realize what is important. Though I
think we should do it by levels and/or prefixes
We also miss some more flexibility with the current tools (not sure if there are even
any), to be able to dynamically change the levels, filter non-related stuff before it gets
written…
But it's great to see some improvement and discussion around this topic.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2:22 p.m.
On Fri, Jul 11, 2014 at 09:13:27AM +0200, Michal Skrivanek wrote:
>
> I have recently introduced a connectivity.log, that tracks changes in
> host network devices. Adding parallel logs for other important
> admin-understandable changes makes sense.
I wonder if this is the right approach…IMHO it's not, as you have to
now take care of the additional file in the context of your code. It's
easy to diverge then.
Why not follow the syslog-like way with config, splitting/limiting files based on e.g.
subpackage prefix, etc…
I am not sure what you suggest. In my implementation, a piece of the
code that wants to log a connectivity-related issue, it asks for
logging.getLogger('connectivity'). The whole idea is that the reader of
this code is not interested on the subpackage or module emitting the
error. He just wants to know that it affects connectivity. In that, it
is similar to Sven and Nir's request - a log file that is more inclining
to the admin, and less to the developer.
I'm sure it was brought up many times before….why don't we just send
it to syslog?:)
IIRC it was striked down since just because it was complex to configure
syslog.conf and/or rsyslog.conf properly. Doing the de-multiplexing of
log messages in /etc/vdsm/logger.conf was much easier.
I think that nowadays, we could just drop a file onto /etc/rsyslog.d/.
<snip>
>
> Vdsm has a setLogLevel; it must have rotton out of disuse. But I'd love
> to see an Engine option for "cluster debug level", that is overridable
> per-host. Each admin should decide what does they care more about:
> debugablility by developers, or the bliss of quiet logs.
per host sounds good for vdsm (btw engine suffers from the same)
I wouldn't bother with UI though….I kind of never understood the tools which have
"enable debug" checkbox in UI but then you anyway have no idea what is it now
creating, what's the impact on the system, where to look for it.
I think debug should remain a purely troubleshooting measure, hence requiring you to log
to host and do something, look/analyze/copy the big dump of info…and then turn it back.
High debug levels typically bring some caveats like less performance…it shouldn't be
something I can blindly check and forgot about it in UI.
I'm thinking about the support engineer. He needs as much logs as the
user can give him. By moving away of default DEBUG level, we are taking
one of his most important tools. The least we can do is to make it easy
on him to re-enable it.
2:27 p.m.
Please keep in mind to also support other syslog daemons than
rsyslog :) Thanks!
Am 11.07.2014 13:22, schrieb Dan Kenigsberg:
I think that nowadays, we could just drop a file onto
/etc/rsyslog.d/.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
2:29 p.m.
----- Original Message -----
From: "Dan Kenigsberg" <danken(a)redhat.com>
To: "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
Cc: devel(a)ovirt.org
Sent: Friday, July 11, 2014 1:22:44 PM
Subject: Re: [ovirt-devel] [vdsm] logging levels and noise in the log
On Fri, Jul 11, 2014 at 09:13:27AM +0200, Michal Skrivanek wrote:
>
> >
> > I have recently introduced a connectivity.log, that tracks changes in
> > host network devices. Adding parallel logs for other important
> > admin-understandable changes makes sense.
>
> I wonder if this is the right approach…IMHO it's not, as you have to
> now take care of the additional file in the context of your code. It's
> easy to diverge then.
> Why not follow the syslog-like way with config, splitting/limiting files
> based on e.g. subpackage prefix, etc…
I am not sure what you suggest. In my implementation, a piece of the
code that wants to log a connectivity-related issue, it asks for
logging.getLogger('connectivity'). The whole idea is that the reader of
this code is not interested on the subpackage or module emitting the
error. He just wants to know that it affects connectivity. In that, it
is similar to Sven and Nir's request - a log file that is more inclining
to the admin, and less to the developer.
>
> I'm sure it was brought up many times before….why don't we just send
> it to syslog?:)
IIRC it was striked down since just because it was complex to configure
syslog.conf and/or rsyslog.conf properly. Doing the de-multiplexing of
log messages in /etc/vdsm/logger.conf was much easier.
I think that nowadays, we could just drop a file onto /etc/rsyslog.d/.
I'd go for this.
<snip>
> >
> > Vdsm has a setLogLevel; it must have rotton out of disuse. But I'd love
> > to see an Engine option for "cluster debug level", that is
overridable
> > per-host. Each admin should decide what does they care more about:
> > debugablility by developers, or the bliss of quiet logs.
>
> per host sounds good for vdsm (btw engine suffers from the same)
> I wouldn't bother with UI though….I kind of never understood the tools
> which have "enable debug" checkbox in UI but then you anyway have no idea
> what is it now creating, what's the impact on the system, where to look
> for it.
> I think debug should remain a purely troubleshooting measure, hence
> requiring you to log to host and do something, look/analyze/copy the big
> dump of info…and then turn it back.
> High debug levels typically bring some caveats like less performance…it
> shouldn't be something I can blindly check and forgot about it in UI.
I'm thinking about the support engineer. He needs as much logs as the
user can give him. By moving away of default DEBUG level, we are taking
one of his most important tools. The least we can do is to make it easy
on him to re-enable it.
I'd have either a verb for vdsmd/supervdsmd or a signal that can be sent
via vdsClient to a running daemon to modify the logging level.
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:21 p.m.
On Jul 11, 2014, at 13:29 , Antoni Segura Puimedon <asegurap(a)redhat.com> wrote:
----- Original Message -----
> From: "Dan Kenigsberg" <danken(a)redhat.com>
> To: "Michal Skrivanek" <michal.skrivanek(a)redhat.com>
> Cc: devel(a)ovirt.org
> Sent: Friday, July 11, 2014 1:22:44 PM
> Subject: Re: [ovirt-devel] [vdsm] logging levels and noise in the log
>
> On Fri, Jul 11, 2014 at 09:13:27AM +0200, Michal Skrivanek wrote:
>>
>>>
>>> I have recently introduced a connectivity.log, that tracks changes in
>>> host network devices. Adding parallel logs for other important
>>> admin-understandable changes makes sense.
>>
>> I wonder if this is the right approach…IMHO it's not, as you have to
>> now take care of the additional file in the context of your code. It's
>> easy to diverge then.
>> Why not follow the syslog-like way with config, splitting/limiting files
>> based on e.g. subpackage prefix, etc…
>
> I am not sure what you suggest. In my implementation, a piece of the
> code that wants to log a connectivity-related issue, it asks for
> logging.getLogger('connectivity'). The whole idea is that the reader of
> this code is not interested on the subpackage or module emitting the
> error. He just wants to know that it affects connectivity. In that, it
> is similar to Sven and Nir's request - a log file that is more inclining
> to the admin, and less to the developer.
I'm not opposed to splitting stuff, only to diverge to different files right away.
I'd prefer all info flows the same way, then do de-multiplexing, to different files or
the same, whatever is required.
As sometimes this admin guy's logs are good enough, but sometimes you need the context
so it's better to have it in the same place (i.e. exactly the same in a different/root
log file)
It's fine you did it once, it's not fine if everyone does that and we end up with
30 distributed files per group and per problem…without a central location how to redirect
things or without a holistic detailed view for real debugging.
anyway, it's a very minor point, nevermind...
>
>>
>> I'm sure it was brought up many times before….why don't we just send
>> it to syslog?:)
>
> IIRC it was striked down since just because it was complex to configure
> syslog.conf and/or rsyslog.conf properly. Doing the de-multiplexing of
> log messages in /etc/vdsm/logger.conf was much easier.
>
> I think that nowadays, we could just drop a file onto /etc/rsyslog.d/.
I'd go for this.
>
> <snip>
>
>>>
>>> Vdsm has a setLogLevel; it must have rotton out of disuse. But I'd love
>>> to see an Engine option for "cluster debug level", that is
overridable
>>> per-host. Each admin should decide what does they care more about:
>>> debugablility by developers, or the bliss of quiet logs.
>>
>> per host sounds good for vdsm (btw engine suffers from the same)
>> I wouldn't bother with UI though….I kind of never understood the tools
>> which have "enable debug" checkbox in UI but then you anyway have no
idea
>> what is it now creating, what's the impact on the system, where to look
>> for it.
>> I think debug should remain a purely troubleshooting measure, hence
>> requiring you to log to host and do something, look/analyze/copy the big
>> dump of info…and then turn it back.
>> High debug levels typically bring some caveats like less performance…it
>> shouldn't be something I can blindly check and forgot about it in UI.
>
> I'm thinking about the support engineer. He needs as much logs as the
> user can give him. By moving away of default DEBUG level, we are taking
> one of his most important tools. The least we can do is to make it easy
> on him to re-enable it.
I'd have either a verb for vdsmd/supervdsmd or a signal that can be sent
via vdsClient to a running daemon to modify the logging level.
I think we'd benefit from some simple set of wrappers/scripts to make it comfortable
Comfortable for support engineers too. I just still want them to at least log on the damn
box, that's the least you can do…:)
(seriously, I'm concerned about too much of an ignorance if it's too simple.
Enabling debug (as well as running logcollector) is a dangerous thing to do)
Thanks,
michal
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
3787
days inactive
3788
days old
11 comments
7 participants
participants (7)
-
Antoni Segura Puimedon -
Dan Kenigsberg -
Francesco Romani -
Martin Sivak -
Michal Skrivanek -
Nir Soffer -
Sven Kieske