4:34 p.m.
Hello,
I remember many months ago a thread where someone complained about too
verbose logging.
Still today in 3.3.1 we have many many DEBUG messages in both vdsm and
engine logs.
Is there already a wiki explaining how to relax logging level?
I think that if we think it is now able to be production ready, this
implies changing default log level.
Right today it is difficult to understand logs because of verbosity,
in my opinion.
Best if in upcoming 3.3.2 release this could take place.
Anyway someone could always leverage the level as he/she needs...
What do you think about it?
Gianluca
6:41 p.m.
Hello Gianluca,
On 12/06/2013 05:34 PM, Gianluca Cecchi wrote:
Hello,
I remember many months ago a thread where someone complained about too
verbose logging.
Still today in 3.3.1 we have many many DEBUG messages in both vdsm and
engine logs.
Is there already a wiki explaining how to relax logging level?
I think that if we think it is now able to be production ready, this
implies changing default log level.
Right today it is difficult to understand logs because of verbosity,
in my opinion.
Best if in upcoming 3.3.2 release this could take place.
Anyway someone could always leverage the level as he/she needs...
What do you think about it?
IMO, looks reasonable. Could you please be more specific and share which
messages are flooding the log files? Would be nice if you open a
bugzilla request [1] based on component (vdsm/ovirt-engine) to get it
tracked.
https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt
--
Cheers
Douglas
5:19 p.m.
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
IMO, looks reasonable. Could you please be more specific and share which
messages are flooding the log files? Would be nice if you open a bugzilla
request [1] based on component (vdsm/ovirt-engine) to get it tracked.
https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt
--
Cheers
Douglas
Reading better, for engine now it is ok and it seems that level is
INFO, no DEBUG messages. Good.
But for vdsm/superdsm, my AIO server that for example today has been
powered on without making anything else than keep an idle windows xp
and an idle fedora 19 vm
has these numbers
in 16 running hours
[root@tekkaman vdsm]# grep "2013-12-07" supervdsm.log | wc -l
11357
[root@tekkaman vdsm]# grep "DEBUG::2013-12-07" supervdsm.log | wc -l
11357
so all and only DEBUG type messages, about one every 6 seconds and
about 10MBytes
[root@tekkaman vdsm]# xzcat vdsm.log-20131207.xz vdsm.log.3.xz
vdsm.log.2.xz vdsm.log.1.xz | grep "2013-12-07" | wc -l
236514
[root@tekkaman vdsm]# xzcat vdsm.log-20131207.xz vdsm.log.3.xz
vdsm.log.2.xz vdsm.log.1.xz | grep "DEBUG::2013-12-07" | wc -l
203038
[root@tekkaman vdsm]# xzcat vdsm.log-20131207.xz vdsm.log.3.xz
vdsm.log.2.xz vdsm.log.1.xz |grep "2013-12-07" | awk -F ":"
'{print
$3}' | sort | uniq -c
203038 DEBUG
1135 ERROR
31870 INFO
471 WARNING
about 4 debug messages per second
BTW: the ERROR ones are for lines such as
Thread-30::ERROR::2013-12-07
23:04:26,939::sdc::137::Storage.StorageDomainCache::(_findDomain)
looking for unfetched domain 0a8035e6-e41d-40ff-a154-e0a374f264b2
Thread-30::ERROR::2013-12-07
23:04:26,939::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
looking for domain 0a8035e6-e41d-40ff-a154-e0a374f264b2
I don't know exactly to what they refer as I don't have an sdc disk
and don't know how to solve....
In my opinion it could be ok to move level to INFO for vdsm too, as
already done for engine.
I don't think it is necessary an rfe for this...
If all is ok no problem, but when and if you have a problem it is
cumbersome to surf through all these DEBUG messages to find relevant
information.
Cheers
Gianluca
3:29 a.m.
On Sun, Dec 8, 2013 at 12:19 AM, Gianluca Cecchi wrote:
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
>
> IMO, looks reasonable. Could you please be more specific and share which
> messages are flooding the log files? Would be nice if you open a bugzilla
> request [1] based on component (vdsm/ovirt-engine) to get it tracked.
>
> https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt
>
> --
> Cheers
> Douglas
Reading better, for engine now it is ok and it seems that level is
INFO, no DEBUG messages. Good.
But for vdsm/superdsm, my AIO server that for example today has been
powered on without making anything else than keep an idle windows xp
and an idle fedora 19 vm
has these numbers
see also here as already leveraged in February for ovirt node and
persist problems
http://lists.ovirt.org/pipermail/users/2013-February/012617.html
It is only a matter of changing default values for /etc/vdsm/logger.conf
In 3.3.1 and still in vdsm as in in 3.3.2 beta we have now
[g.cecchi@tekkaman ~]$ grep -B1 DEBUG /etc/vdsm/logger.conf
[logger_root]
level=DEBUG
--
[logger_vds]
level=DEBUG
--
[logger_Storage]
level=DEBUG
--
filters=storage.misc.TracebackRepeatFilter
level=DEBUG
Changing these lines in my environment to INFO kept off the DEBUG
messages with about one INFO message per second.
Or we could provide a page on the wiki and/or release notes where we
explain how to change logging levels and files involved if you think
that DEBUG for vdsmd is still the way to have.
Gianluca
9:06 a.m.
On Sun, Dec 08, 2013 at 12:19:10AM +0100, Gianluca Cecchi wrote:
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
BTW: the ERROR ones are for lines such as
Thread-30::ERROR::2013-12-07
23:04:26,939::sdc::137::Storage.StorageDomainCache::(_findDomain)
looking for unfetched domain 0a8035e6-e41d-40ff-a154-e0a374f264b2
Thread-30::ERROR::2013-12-07
23:04:26,939::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
looking for domain 0a8035e6-e41d-40ff-a154-e0a374f264b2
I don't know exactly to what they refer as I don't have an sdc disk
and don't know how to solve....
This log line indicates a serious error if that domain has an iSCSI path to it, and is
quite bogus if it's pure FC.
We need to solve this duality in one way or another.
http://gerrit.ovirt.org/#/c/21597/ for more reference.
10:56 a.m.
On Sun, Dec 8, 2013 at 4:06 PM, Dan Kenigsberg wrote:
On Sun, Dec 08, 2013 at 12:19:10AM +0100, Gianluca Cecchi wrote:
> On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
>
>
> BTW: the ERROR ones are for lines such as
> Thread-30::ERROR::2013-12-07
> 23:04:26,939::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 0a8035e6-e41d-40ff-a154-e0a374f264b2
> Thread-30::ERROR::2013-12-07
> 23:04:26,939::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 0a8035e6-e41d-40ff-a154-e0a374f264b2
> I don't know exactly to what they refer as I don't have an sdc disk
> and don't know how to solve....
This log line indicates a serious error if that domain has an iSCSI path to it, and is
quite bogus if it's pure FC.
We need to solve this duality in one way or another.
http://gerrit.ovirt.org/#/c/21597/ for more reference.
Thanks for the info.
Actually in my case we are talking about an AIO install with local on
host datacenter and local on host storage pointing to /DATA directory
(that is under / filesystem).
I then added another local on host storage domain pointing to another
directory (/data/DATA2 where /data is another filesystem mount point)
Is it going to be addressed in 3.3.2 beta and/or final?
Gianluca
4032
days inactive
4034
days old
5 comments
3 participants
participants (3)
-
Dan Kenigsberg -
Douglas Schilling Landgraf -
Gianluca Cecchi