[Users] avoid debug messages or explain better how to change log level
Hello, I remember many months ago a thread where someone complained about too verbose logging. Still today in 3.3.1 we have many many DEBUG messages in both vdsm and engine logs. Is there already a wiki explaining how to relax logging level? I think that if we think it is now able to be production ready, this implies changing default log level. Right today it is difficult to understand logs because of verbosity, in my opinion. Best if in upcoming 3.3.2 release this could take place. Anyway someone could always leverage the level as he/she needs... What do you think about it? Gianluca
Hello Gianluca, On 12/06/2013 05:34 PM, Gianluca Cecchi wrote:
Hello, I remember many months ago a thread where someone complained about too verbose logging. Still today in 3.3.1 we have many many DEBUG messages in both vdsm and engine logs. Is there already a wiki explaining how to relax logging level? I think that if we think it is now able to be production ready, this implies changing default log level. Right today it is difficult to understand logs because of verbosity, in my opinion. Best if in upcoming 3.3.2 release this could take place. Anyway someone could always leverage the level as he/she needs...
What do you think about it?
IMO, looks reasonable. Could you please be more specific and share which messages are flooding the log files? Would be nice if you open a bugzilla request [1] based on component (vdsm/ovirt-engine) to get it tracked. https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt -- Cheers Douglas
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
IMO, looks reasonable. Could you please be more specific and share which messages are flooding the log files? Would be nice if you open a bugzilla request [1] based on component (vdsm/ovirt-engine) to get it tracked.
https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt
-- Cheers Douglas
Reading better, for engine now it is ok and it seems that level is INFO, no DEBUG messages. Good. But for vdsm/superdsm, my AIO server that for example today has been powered on without making anything else than keep an idle windows xp and an idle fedora 19 vm has these numbers in 16 running hours [root@tekkaman vdsm]# grep "2013-12-07" supervdsm.log | wc -l 11357 [root@tekkaman vdsm]# grep "DEBUG::2013-12-07" supervdsm.log | wc -l 11357 so all and only DEBUG type messages, about one every 6 seconds and about 10MBytes [root@tekkaman vdsm]# xzcat vdsm.log-20131207.xz vdsm.log.3.xz vdsm.log.2.xz vdsm.log.1.xz | grep "2013-12-07" | wc -l 236514 [root@tekkaman vdsm]# xzcat vdsm.log-20131207.xz vdsm.log.3.xz vdsm.log.2.xz vdsm.log.1.xz | grep "DEBUG::2013-12-07" | wc -l 203038 [root@tekkaman vdsm]# xzcat vdsm.log-20131207.xz vdsm.log.3.xz vdsm.log.2.xz vdsm.log.1.xz |grep "2013-12-07" | awk -F ":" '{print $3}' | sort | uniq -c 203038 DEBUG 1135 ERROR 31870 INFO 471 WARNING about 4 debug messages per second BTW: the ERROR ones are for lines such as Thread-30::ERROR::2013-12-07 23:04:26,939::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 0a8035e6-e41d-40ff-a154-e0a374f264b2 Thread-30::ERROR::2013-12-07 23:04:26,939::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 0a8035e6-e41d-40ff-a154-e0a374f264b2 I don't know exactly to what they refer as I don't have an sdc disk and don't know how to solve.... In my opinion it could be ok to move level to INFO for vdsm too, as already done for engine. I don't think it is necessary an rfe for this... If all is ok no problem, but when and if you have a problem it is cumbersome to surf through all these DEBUG messages to find relevant information. Cheers Gianluca
On Sun, Dec 8, 2013 at 12:19 AM, Gianluca Cecchi wrote:
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
IMO, looks reasonable. Could you please be more specific and share which messages are flooding the log files? Would be nice if you open a bugzilla request [1] based on component (vdsm/ovirt-engine) to get it tracked.
https://bugzilla.redhat.com/enter_bug.cgi?product=oVirt
-- Cheers Douglas
Reading better, for engine now it is ok and it seems that level is INFO, no DEBUG messages. Good. But for vdsm/superdsm, my AIO server that for example today has been powered on without making anything else than keep an idle windows xp and an idle fedora 19 vm has these numbers
see also here as already leveraged in February for ovirt node and persist problems http://lists.ovirt.org/pipermail/users/2013-February/012617.html It is only a matter of changing default values for /etc/vdsm/logger.conf In 3.3.1 and still in vdsm as in in 3.3.2 beta we have now [g.cecchi@tekkaman ~]$ grep -B1 DEBUG /etc/vdsm/logger.conf [logger_root] level=DEBUG -- [logger_vds] level=DEBUG -- [logger_Storage] level=DEBUG -- filters=storage.misc.TracebackRepeatFilter level=DEBUG Changing these lines in my environment to INFO kept off the DEBUG messages with about one INFO message per second. Or we could provide a page on the wiki and/or release notes where we explain how to change logging levels and files involved if you think that DEBUG for vdsmd is still the way to have. Gianluca
On Sun, Dec 08, 2013 at 12:19:10AM +0100, Gianluca Cecchi wrote:
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
BTW: the ERROR ones are for lines such as Thread-30::ERROR::2013-12-07 23:04:26,939::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 0a8035e6-e41d-40ff-a154-e0a374f264b2 Thread-30::ERROR::2013-12-07 23:04:26,939::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 0a8035e6-e41d-40ff-a154-e0a374f264b2 I don't know exactly to what they refer as I don't have an sdc disk and don't know how to solve....
This log line indicates a serious error if that domain has an iSCSI path to it, and is quite bogus if it's pure FC. We need to solve this duality in one way or another. http://gerrit.ovirt.org/#/c/21597/ for more reference.
On Sun, Dec 8, 2013 at 4:06 PM, Dan Kenigsberg wrote:
On Sun, Dec 08, 2013 at 12:19:10AM +0100, Gianluca Cecchi wrote:
On Sun, Dec 8, 2013 at 1:41 AM, Douglas Schilling Landgraf wrote:
BTW: the ERROR ones are for lines such as Thread-30::ERROR::2013-12-07 23:04:26,939::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 0a8035e6-e41d-40ff-a154-e0a374f264b2 Thread-30::ERROR::2013-12-07 23:04:26,939::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 0a8035e6-e41d-40ff-a154-e0a374f264b2 I don't know exactly to what they refer as I don't have an sdc disk and don't know how to solve....
This log line indicates a serious error if that domain has an iSCSI path to it, and is quite bogus if it's pure FC.
We need to solve this duality in one way or another.
http://gerrit.ovirt.org/#/c/21597/ for more reference.
Thanks for the info. Actually in my case we are talking about an AIO install with local on host datacenter and local on host storage pointing to /DATA directory (that is under / filesystem). I then added another local on host storage domain pointing to another directory (/data/DATA2 where /data is another filesystem mount point) Is it going to be addressed in 3.3.2 beta and/or final? Gianluca
participants (3)
-
Dan Kenigsberg -
Douglas Schilling Landgraf -
Gianluca Cecchi