Tracking down a vdsm memory leak
I have a problem with a memory leak in vdsm. I have a dev cluster that right now is: - two nodes - CentOS 7.4 (up to date) - oVirt 4.2.2 (installed as 3.5 and upgraded version by version) - hosted engine (no other running VM at the moment) - iSCSI storage I have a script that writes the vdsm RSS to a file every five minutes, and on the node holding the hosted engine, vdsm RSS grows around 300-1500KB every snapshot. I maintain several oVirt clusters for others, and they all seem to have this problem. The production clusters are all still on oVirt 4.1, but they all have this problem too, so I guess it is something about how I set them up? On a couple I just checked, the vdsm RSS is over 1G. Any tips on instrumenting vdsm to track this down? I am unfortunately only passingly familiar with python (I can make small changes, but not knowledgeable enough to figure this out). -- Chris Adams <cma@cmadams.net>
On 7 May 2018, at 17:46, Chris Adams <cma@cmadams.net> wrote:
I have a problem with a memory leak in vdsm. I have a dev cluster that right now is:
- two nodes - CentOS 7.4 (up to date) - oVirt 4.2.2 (installed as 3.5 and upgraded version by version) - hosted engine (no other running VM at the moment) - iSCSI storage
I have a script that writes the vdsm RSS to a file every five minutes, and on the node holding the hosted engine, vdsm RSS grows around 300-1500KB every snapshot.
so it’s not happening on other nodes which are empty, and it’s not happening when other nodes are running some VM, as long as it is not the HE VM?
I maintain several oVirt clusters for others, and they all seem to have this problem. The production clusters are all still on oVirt 4.1, but they all have this problem too, so I guess it is something about how I set them up? On a couple I just checked, the vdsm RSS is over 1G.
Any tips on instrumenting vdsm to track this down? I am unfortunately only passingly familiar with python (I can make small changes, but not knowledgeable enough to figure this out).
-- Chris Adams <cma@cmadams.net> _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Once upon a time, Michal Skrivanek <michal.skrivanek@redhat.com> said:
so it’s not happening on other nodes which are empty, and it’s not happening when other nodes are running some VM, as long as it is not the HE VM?
Correct. I just simplified in my dev setup to not running any other VMs to make is easier to verify (nobody is trying to use any of the dev VMs right now). -- Chris Adams <cma@cmadams.net>
On Mon, May 7, 2018 at 6:48 PM Chris Adams <cma@cmadams.net> wrote:
I have a problem with a memory leak in vdsm. I have a dev cluster that right now is:
- two nodes - CentOS 7.4 (up to date) - oVirt 4.2.2 (installed as 3.5 and upgraded version by version) - hosted engine (no other running VM at the moment) - iSCSI storage
I have a script that writes the vdsm RSS to a file every five minutes, and on the node holding the hosted engine, vdsm RSS grows around 300-1500KB every snapshot.
I maintain several oVirt clusters for others, and they all seem to have this problem. The production clusters are all still on oVirt 4.1, but they all have this problem too, so I guess it is something about how I set them up? On a couple I just checked, the vdsm RSS is over 1G.
Any tips on instrumenting vdsm to track this down? I am unfortunately only passingly familiar with python (I can make small changes, but not knowledgeable enough to figure this out).
To debug these issues, you should enable the health monitor by creating a drop-in configuration file: $ cat /etc/vdsm/vdsm.conf.d/health.conf [devel] health_monitor_enable = true And restart vdsm to start the health monitor. The health logs are using DEBUG level so you need to enable DEBUG level for the "health" logger. You can do this with: $ vdsm-client Host setLogLevel level=DEBUG name=health Or by adding new logger configuration to /etc/vdsm/logger.conf: 1. add health logger to [loggers] [loggers] keys=root,vds,storage,virt,ovirt_hosted_engine_ha,ovirt_hosted_engine_ha_config,IOProcess,devel,health 2. add [logger_health] section [logger_health] level=DEBUG handlers=logthread qualname=health propagate=0 Finally post here the [health] logs from vdsm.log. Nir
On Tue, May 8, 2018 at 9:16 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Mon, May 7, 2018 at 6:48 PM Chris Adams <cma@cmadams.net> wrote:
I have a problem with a memory leak in vdsm. I have a dev cluster that right now is:
- two nodes - CentOS 7.4 (up to date) - oVirt 4.2.2 (installed as 3.5 and upgraded version by version) - hosted engine (no other running VM at the moment) - iSCSI storage
I have a script that writes the vdsm RSS to a file every five minutes, and on the node holding the hosted engine, vdsm RSS grows around 300-1500KB every snapshot.
I maintain several oVirt clusters for others, and they all seem to have this problem. The production clusters are all still on oVirt 4.1, but they all have this problem too, so I guess it is something about how I set them up? On a couple I just checked, the vdsm RSS is over 1G.
Any tips on instrumenting vdsm to track this down? I am unfortunately only passingly familiar with python (I can make small changes, but not knowledgeable enough to figure this out).
To debug these issues, you should enable the health monitor by creating a drop-in configuration file:
$ cat /etc/vdsm/vdsm.conf.d/health.conf [devel] health_monitor_enable = true
And restart vdsm to start the health monitor.
The health logs are using DEBUG level so you need to enable DEBUG level for the "health" logger. You can do this with:
$ vdsm-client Host setLogLevel level=DEBUG name=health
Or by adding new logger configuration to /etc/vdsm/logger.conf:
1. add health logger to [loggers]
[loggers]
keys=root,vds,storage,virt,ovirt_hosted_engine_ha,ovirt_hosted_engine_ha_config,IOProcess,devel,health
2. add [logger_health] section
[logger_health] level=DEBUG handlers=logthread qualname=health propagate=0
Finally post here the [health] logs from vdsm.log.
Here are example logs on a host running 4.2.3 without hosted engine: # grep '(health)' /var/log/vdsm/vdsm.log 2018-05-08 10:54:29,424+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:54:29,451+0300 DEBUG (health) [health] Collected 4710 objects (health:97) 2018-05-08 10:54:29,451+0300 DEBUG (health) [health] user=2.22%, sys=0.92%, rss=84192 kB (+23436), threads=70 (health:122) 2018-05-08 10:55:29,451+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:55:29,472+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 10:55:29,474+0300 DEBUG (health) [health] user=0.60%, sys=0.38%, rss=84236 kB (+44), threads=62 (health:122) 2018-05-08 10:56:29,475+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:56:29,496+0300 DEBUG (health) [health] Collected 48 objects (health:97) 2018-05-08 10:56:29,496+0300 DEBUG (health) [health] user=0.58%, sys=0.42%, rss=84444 kB (+208), threads=62 (health:122) 2018-05-08 10:57:29,497+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:57:29,521+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 10:57:29,521+0300 DEBUG (health) [health] user=0.58%, sys=0.42%, rss=84452 kB (+8), threads=62 (health:122) 2018-05-08 10:58:29,522+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:58:29,545+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 10:58:29,545+0300 DEBUG (health) [health] user=0.60%, sys=0.42%, rss=84500 kB (+48), threads=62 (health:122) 2018-05-08 10:59:29,546+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:59:29,569+0300 DEBUG (health) [health] Collected 137 objects (health:97) 2018-05-08 10:59:29,570+0300 DEBUG (health) [health] user=0.70%, sys=0.48%, rss=84876 kB (+376), threads=69 (health:122) 2018-05-08 11:00:29,570+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 11:00:29,593+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 11:00:29,594+0300 DEBUG (health) [health] user=0.60%, sys=0.43%, rss=84744 kB (-132), threads=62 (health:122) 2018-05-08 11:01:29,594+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 11:01:29,617+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 11:01:29,617+0300 DEBUG (health) [health] user=0.65%, sys=0.40%, rss=84748 kB (+4), threads=62 (health:122) 2018-05-08 11:02:29,618+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 11:02:29,641+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 11:02:29,641+0300 DEBUG (health) [health] user=0.57%, sys=0.40%, rss=84748 kB (+0), threads=62 (health:122) 2018-05-08 11:03:29,642+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 11:03:29,665+0300 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 11:03:29,665+0300 DEBUG (health) [health] user=0.60%, sys=0.43%, rss=84812 kB (+64), threads=62 (health:122) 2018-05-08 11:04:29,666+0300 DEBUG (health) [health] Checking health (health:90) 2018-05-08 11:04:29,690+0300 DEBUG (health) [health] Collected 137 objects (health:97) 2018-05-08 11:04:29,690+0300 DEBUG (health) [health] user=0.73%, sys=0.47%, rss=85008 kB (+196), threads=69 (health:122) Health stats are also reported using the configured metrics collector, (see [metrics] section in vdsm.conf) using these names: hosts.vdsm.gc.uncollectable hosts.vdsm.cpu.user_pct hosts.vdsm.cpu.sys_pct hosts.vdsm.cpu.memory.rss hosts.vdsm.threads_count Nir
Here's the log after the first couple of hours. It is growing pretty steadliy. 2018-05-08 07:49:02,422-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:49:02,453-0500 DEBUG (health) [health] Collected 1307 objects (health:97) 2018-05-08 07:49:02,454-0500 DEBUG (health) [health] user=3.83%, sys=1.27%, rss=79344 kB (+14664), threads=46 (health:122) 2018-05-08 07:50:02,454-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:50:02,479-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 07:50:02,480-0500 DEBUG (health) [health] user=1.18%, sys=0.47%, rss=85468 kB (+6124), threads=46 (health:122) 2018-05-08 07:51:02,481-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:51:02,507-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 07:51:02,507-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=87716 kB (+2248), threads=46 (health:122) 2018-05-08 07:52:02,508-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:52:02,534-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 07:52:02,535-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=90176 kB (+2460), threads=46 (health:122) 2018-05-08 07:53:02,536-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:53:02,561-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 07:53:02,561-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=90332 kB (+156), threads=46 (health:122) 2018-05-08 07:54:02,562-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:54:02,589-0500 DEBUG (health) [health] Collected 110 objects (health:97) 2018-05-08 07:54:02,589-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=90472 kB (+140), threads=46 (health:122) 2018-05-08 07:55:02,590-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:55:02,615-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 07:55:02,616-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=90560 kB (+88), threads=46 (health:122) 2018-05-08 07:56:02,616-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:56:02,645-0500 DEBUG (health) [health] Collected 90 objects (health:97) 2018-05-08 07:56:02,645-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=90944 kB (+384), threads=46 (health:122) 2018-05-08 07:57:02,646-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:57:02,672-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 07:57:02,673-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=91076 kB (+132), threads=46 (health:122) 2018-05-08 07:58:02,674-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:58:02,700-0500 DEBUG (health) [health] Collected 54 objects (health:97) 2018-05-08 07:58:02,703-0500 DEBUG (health) [health] user=1.22%, sys=0.42%, rss=91500 kB (+424), threads=46 (health:122) 2018-05-08 07:59:02,703-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:59:02,730-0500 DEBUG (health) [health] Collected 110 objects (health:97) 2018-05-08 07:59:02,730-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=91584 kB (+84), threads=46 (health:122) 2018-05-08 08:00:02,731-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:00:02,757-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:00:02,759-0500 DEBUG (health) [health] user=1.25%, sys=0.43%, rss=91696 kB (+112), threads=46 (health:122) 2018-05-08 08:01:02,760-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:01:02,789-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:01:02,790-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=91760 kB (+64), threads=46 (health:122) 2018-05-08 08:02:02,791-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:02:02,822-0500 DEBUG (health) [health] Collected 90 objects (health:97) 2018-05-08 08:02:02,823-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=92084 kB (+324), threads=46 (health:122) 2018-05-08 08:03:02,824-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:03:02,851-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 08:03:02,851-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=92308 kB (+224), threads=46 (health:122) 2018-05-08 08:04:02,853-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:04:02,891-0500 DEBUG (health) [health] Collected 99 objects (health:97) 2018-05-08 08:04:02,892-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=92652 kB (+344), threads=46 (health:122) 2018-05-08 08:05:02,893-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:05:02,921-0500 DEBUG (health) [health] Collected 51 objects (health:97) 2018-05-08 08:05:02,922-0500 DEBUG (health) [health] user=1.23%, sys=0.43%, rss=92772 kB (+120), threads=46 (health:122) 2018-05-08 08:06:02,922-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:06:02,948-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 08:06:02,949-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=92868 kB (+96), threads=46 (health:122) 2018-05-08 08:07:02,949-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:07:02,976-0500 DEBUG (health) [health] Collected 87 objects (health:97) 2018-05-08 08:07:02,979-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=93236 kB (+368), threads=46 (health:122) 2018-05-08 08:08:02,984-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:08:03,019-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 08:08:03,020-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=93436 kB (+200), threads=46 (health:122) 2018-05-08 08:09:03,021-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:09:03,048-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:09:03,048-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=93776 kB (+340), threads=46 (health:122) 2018-05-08 08:10:03,049-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:10:03,088-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 08:10:03,088-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=93836 kB (+60), threads=46 (health:122) 2018-05-08 08:11:03,089-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:11:03,115-0500 DEBUG (health) [health] Collected 54 objects (health:97) 2018-05-08 08:11:03,116-0500 DEBUG (health) [health] user=1.15%, sys=0.48%, rss=93936 kB (+100), threads=46 (health:122) 2018-05-08 08:12:03,117-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:12:03,142-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:12:03,143-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=94052 kB (+116), threads=46 (health:122) 2018-05-08 08:13:03,143-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:13:03,169-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 08:13:03,169-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=94200 kB (+148), threads=46 (health:122) 2018-05-08 08:14:03,170-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:14:03,199-0500 DEBUG (health) [health] Collected 131 objects (health:97) 2018-05-08 08:14:03,200-0500 DEBUG (health) [health] user=1.27%, sys=0.53%, rss=94576 kB (+376), threads=46 (health:122) 2018-05-08 08:15:03,200-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:15:03,228-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:15:03,232-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=94660 kB (+84), threads=46 (health:122) 2018-05-08 08:16:03,232-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:16:03,258-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 08:16:03,261-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=95012 kB (+352), threads=46 (health:122) 2018-05-08 08:17:03,262-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:17:03,293-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 08:17:03,294-0500 DEBUG (health) [health] user=1.18%, sys=0.50%, rss=95076 kB (+64), threads=46 (health:122) 2018-05-08 08:18:03,295-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:18:03,320-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:18:03,323-0500 DEBUG (health) [health] user=1.20%, sys=0.45%, rss=95168 kB (+92), threads=46 (health:122) 2018-05-08 08:19:03,324-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:19:03,350-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 08:19:03,350-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=95248 kB (+80), threads=46 (health:122) 2018-05-08 08:20:03,351-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:20:03,376-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 08:20:03,377-0500 DEBUG (health) [health] user=1.28%, sys=0.47%, rss=95272 kB (+24), threads=46 (health:122) 2018-05-08 08:21:03,377-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:21:03,420-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 08:21:03,420-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=95664 kB (+392), threads=46 (health:122) 2018-05-08 08:22:03,421-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:22:03,447-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:22:03,449-0500 DEBUG (health) [health] user=1.27%, sys=0.52%, rss=95800 kB (+136), threads=46 (health:122) 2018-05-08 08:23:03,449-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:23:03,477-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 08:23:03,477-0500 DEBUG (health) [health] user=1.28%, sys=0.52%, rss=96216 kB (+416), threads=46 (health:122) 2018-05-08 08:24:03,478-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:24:03,505-0500 DEBUG (health) [health] Collected 87 objects (health:97) 2018-05-08 08:24:03,505-0500 DEBUG (health) [health] user=1.35%, sys=0.52%, rss=96748 kB (+532), threads=46 (health:122) 2018-05-08 08:25:03,506-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:25:03,533-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:25:03,534-0500 DEBUG (health) [health] user=1.28%, sys=0.47%, rss=96884 kB (+136), threads=46 (health:122) 2018-05-08 08:26:03,534-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:26:03,560-0500 DEBUG (health) [health] Collected 63 objects (health:97) 2018-05-08 08:26:03,561-0500 DEBUG (health) [health] user=1.30%, sys=0.43%, rss=97640 kB (+756), threads=46 (health:122) 2018-05-08 08:27:03,562-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:27:03,591-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 08:27:03,593-0500 DEBUG (health) [health] user=1.25%, sys=0.52%, rss=97792 kB (+152), threads=46 (health:122) 2018-05-08 08:28:03,593-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:28:03,619-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 08:28:03,621-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=97904 kB (+112), threads=46 (health:122) 2018-05-08 08:29:03,621-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:29:03,648-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:29:03,648-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=97976 kB (+72), threads=46 (health:122) 2018-05-08 08:30:03,649-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:30:03,677-0500 DEBUG (health) [health] Collected 44 objects (health:97) 2018-05-08 08:30:03,677-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=98016 kB (+40), threads=46 (health:122) 2018-05-08 08:31:03,678-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:31:03,706-0500 DEBUG (health) [health] Collected 149 objects (health:97) 2018-05-08 08:31:03,706-0500 DEBUG (health) [health] user=1.23%, sys=0.42%, rss=98372 kB (+356), threads=46 (health:122) 2018-05-08 08:32:03,707-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:32:03,733-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:32:03,734-0500 DEBUG (health) [health] user=1.23%, sys=0.43%, rss=98456 kB (+84), threads=46 (health:122) 2018-05-08 08:33:03,734-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:33:03,760-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:33:03,762-0500 DEBUG (health) [health] user=1.20%, sys=0.45%, rss=98516 kB (+60), threads=46 (health:122) 2018-05-08 08:34:03,762-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:34:03,789-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:34:03,789-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=98584 kB (+68), threads=46 (health:122) 2018-05-08 08:35:03,790-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:35:03,816-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:35:03,818-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=98632 kB (+48), threads=46 (health:122) 2018-05-08 08:36:03,818-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:36:03,851-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:36:03,852-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=98944 kB (+312), threads=46 (health:122) 2018-05-08 08:37:03,854-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:37:03,881-0500 DEBUG (health) [health] Collected 188 objects (health:97) 2018-05-08 08:37:03,882-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=98992 kB (+48), threads=46 (health:122) 2018-05-08 08:38:03,882-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:38:03,911-0500 DEBUG (health) [health] Collected 167 objects (health:97) 2018-05-08 08:38:03,911-0500 DEBUG (health) [health] user=1.20%, sys=0.47%, rss=99076 kB (+84), threads=46 (health:122) 2018-05-08 08:39:03,912-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:39:03,939-0500 DEBUG (health) [health] Collected 111 objects (health:97) 2018-05-08 08:39:03,939-0500 DEBUG (health) [health] user=1.20%, sys=0.52%, rss=99408 kB (+332), threads=46 (health:122) 2018-05-08 08:40:03,940-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:40:03,967-0500 DEBUG (health) [health] Collected 107 objects (health:97) 2018-05-08 08:40:03,968-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=99492 kB (+84), threads=46 (health:122) 2018-05-08 08:41:03,969-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:41:03,997-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 08:41:03,997-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=99596 kB (+104), threads=46 (health:122) 2018-05-08 08:42:03,998-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:42:04,026-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 08:42:04,027-0500 DEBUG (health) [health] user=1.23%, sys=0.43%, rss=99684 kB (+88), threads=46 (health:122) 2018-05-08 08:43:04,027-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:43:04,055-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 08:43:04,056-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=100052 kB (+368), threads=46 (health:122) 2018-05-08 08:44:04,057-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:44:04,100-0500 DEBUG (health) [health] Collected 83 objects (health:97) 2018-05-08 08:44:04,101-0500 DEBUG (health) [health] user=1.28%, sys=0.45%, rss=100168 kB (+116), threads=46 (health:122) 2018-05-08 08:45:04,101-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:45:04,129-0500 DEBUG (health) [health] Collected 19 objects (health:97) 2018-05-08 08:45:04,129-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=100244 kB (+76), threads=46 (health:122) 2018-05-08 08:46:04,130-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:46:04,157-0500 DEBUG (health) [health] Collected 104 objects (health:97) 2018-05-08 08:46:04,157-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=100340 kB (+96), threads=46 (health:122) 2018-05-08 08:47:04,158-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:47:04,186-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 08:47:04,186-0500 DEBUG (health) [health] user=1.27%, sys=0.52%, rss=100672 kB (+332), threads=46 (health:122) 2018-05-08 08:48:04,187-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:48:04,216-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:48:04,216-0500 DEBUG (health) [health] user=1.18%, sys=0.47%, rss=100740 kB (+68), threads=46 (health:122) 2018-05-08 08:49:04,217-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:49:04,246-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 08:49:04,246-0500 DEBUG (health) [health] user=1.32%, sys=0.50%, rss=100828 kB (+88), threads=46 (health:122) 2018-05-08 08:50:04,247-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:50:04,275-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:50:04,275-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=101196 kB (+368), threads=46 (health:122) 2018-05-08 08:51:04,276-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:51:04,305-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 08:51:04,322-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=101284 kB (+88), threads=46 (health:122) 2018-05-08 08:52:04,323-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:52:04,353-0500 DEBUG (health) [health] Collected 122 objects (health:97) 2018-05-08 08:52:04,353-0500 DEBUG (health) [health] user=1.22%, sys=0.45%, rss=101632 kB (+348), threads=46 (health:122) 2018-05-08 08:53:04,354-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:53:04,387-0500 DEBUG (health) [health] Collected 93 objects (health:97) 2018-05-08 08:53:04,387-0500 DEBUG (health) [health] user=1.30%, sys=0.48%, rss=101724 kB (+92), threads=46 (health:122) 2018-05-08 08:54:04,388-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:54:04,416-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 08:54:04,416-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=101800 kB (+76), threads=46 (health:122) 2018-05-08 08:55:04,421-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:55:04,449-0500 DEBUG (health) [health] Collected 24 objects (health:97) 2018-05-08 08:55:04,449-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=101848 kB (+48), threads=46 (health:122) 2018-05-08 08:56:04,450-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:56:04,479-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:56:04,480-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=102192 kB (+344), threads=46 (health:122) 2018-05-08 08:57:04,481-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:57:04,509-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:57:04,510-0500 DEBUG (health) [health] user=1.27%, sys=0.45%, rss=102256 kB (+64), threads=46 (health:122) 2018-05-08 08:58:04,510-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:58:04,555-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 08:58:04,556-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=102380 kB (+124), threads=46 (health:122) 2018-05-08 08:59:04,557-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:59:04,585-0500 DEBUG (health) [health] Collected 115 objects (health:97) 2018-05-08 08:59:04,586-0500 DEBUG (health) [health] user=1.30%, sys=0.53%, rss=102796 kB (+416), threads=46 (health:122) 2018-05-08 09:00:04,586-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:00:04,616-0500 DEBUG (health) [health] Collected 108 objects (health:97) 2018-05-08 09:00:04,617-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=102944 kB (+148), threads=46 (health:122) 2018-05-08 09:01:04,617-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:01:04,651-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 09:01:04,652-0500 DEBUG (health) [health] user=1.27%, sys=0.43%, rss=103048 kB (+104), threads=46 (health:122) 2018-05-08 09:02:04,653-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:02:04,682-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 09:02:04,683-0500 DEBUG (health) [health] user=1.28%, sys=0.42%, rss=103388 kB (+340), threads=46 (health:122) 2018-05-08 09:03:04,684-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:03:04,713-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 09:03:04,714-0500 DEBUG (health) [health] user=1.25%, sys=0.43%, rss=103488 kB (+100), threads=46 (health:122) 2018-05-08 09:04:04,715-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:04:04,743-0500 DEBUG (health) [health] Collected 64 objects (health:97) 2018-05-08 09:04:04,744-0500 DEBUG (health) [health] user=1.27%, sys=0.50%, rss=103592 kB (+104), threads=46 (health:122) 2018-05-08 09:05:04,744-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:05:04,775-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 09:05:04,776-0500 DEBUG (health) [health] user=1.22%, sys=0.45%, rss=103912 kB (+320), threads=46 (health:122) 2018-05-08 09:06:04,776-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:06:04,804-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 09:06:04,804-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=104016 kB (+104), threads=46 (health:122) 2018-05-08 09:07:04,805-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:07:04,832-0500 DEBUG (health) [health] Collected 104 objects (health:97) 2018-05-08 09:07:04,833-0500 DEBUG (health) [health] user=1.27%, sys=0.45%, rss=104104 kB (+88), threads=46 (health:122) 2018-05-08 09:08:04,834-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:08:04,861-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:08:04,862-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=104200 kB (+96), threads=46 (health:122) 2018-05-08 09:09:04,863-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:09:04,890-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:09:04,891-0500 DEBUG (health) [health] user=1.28%, sys=0.53%, rss=104584 kB (+384), threads=46 (health:122) 2018-05-08 09:10:04,891-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:10:04,920-0500 DEBUG (health) [health] Collected 111 objects (health:97) 2018-05-08 09:10:04,921-0500 DEBUG (health) [health] user=1.27%, sys=0.50%, rss=104692 kB (+108), threads=46 (health:122) 2018-05-08 09:11:04,922-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:11:04,950-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 09:11:04,951-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=104744 kB (+52), threads=46 (health:122) 2018-05-08 09:12:04,951-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:12:04,980-0500 DEBUG (health) [health] Collected 52 objects (health:97) 2018-05-08 09:12:04,997-0500 DEBUG (health) [health] user=1.20%, sys=0.52%, rss=105092 kB (+348), threads=46 (health:122) 2018-05-08 09:13:04,998-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:13:05,026-0500 DEBUG (health) [health] Collected 67 objects (health:97) 2018-05-08 09:13:05,027-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=105180 kB (+88), threads=46 (health:122) 2018-05-08 09:14:05,027-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:14:05,055-0500 DEBUG (health) [health] Collected 131 objects (health:97) 2018-05-08 09:14:05,056-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=105536 kB (+356), threads=46 (health:122) 2018-05-08 09:15:05,056-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:15:05,085-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:15:05,086-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=105636 kB (+100), threads=46 (health:122) 2018-05-08 09:16:05,087-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:16:05,115-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 09:16:05,116-0500 DEBUG (health) [health] user=1.20%, sys=0.47%, rss=105724 kB (+88), threads=46 (health:122) 2018-05-08 09:17:05,116-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:17:05,144-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:17:05,146-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=106036 kB (+312), threads=46 (health:122) 2018-05-08 09:18:05,146-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:18:05,174-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:18:05,175-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=106104 kB (+68), threads=46 (health:122) 2018-05-08 09:19:05,175-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:19:05,204-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:19:05,204-0500 DEBUG (health) [health] user=1.20%, sys=0.53%, rss=106228 kB (+124), threads=46 (health:122) 2018-05-08 09:20:05,205-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:20:05,233-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 09:20:05,234-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=106292 kB (+64), threads=46 (health:122) 2018-05-08 09:21:05,235-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:21:05,263-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 09:21:05,263-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=106676 kB (+384), threads=46 (health:122) 2018-05-08 09:22:05,264-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:22:05,292-0500 DEBUG (health) [health] Collected 143 objects (health:97) 2018-05-08 09:22:05,293-0500 DEBUG (health) [health] user=1.27%, sys=0.45%, rss=107008 kB (+332), threads=46 (health:122) 2018-05-08 09:23:05,293-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:23:05,322-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 09:23:05,324-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=107072 kB (+64), threads=46 (health:122) 2018-05-08 09:24:05,324-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:24:05,353-0500 DEBUG (health) [health] Collected 127 objects (health:97) 2018-05-08 09:24:05,353-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=107192 kB (+120), threads=46 (health:122) 2018-05-08 09:25:05,354-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:25:05,381-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 09:25:05,383-0500 DEBUG (health) [health] user=1.18%, sys=0.53%, rss=107356 kB (+164), threads=46 (health:122) 2018-05-08 09:26:05,383-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:26:05,411-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 09:26:05,412-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=107708 kB (+352), threads=46 (health:122) 2018-05-08 09:27:05,412-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:27:05,440-0500 DEBUG (health) [health] Collected 125 objects (health:97) 2018-05-08 09:27:05,441-0500 DEBUG (health) [health] user=1.27%, sys=0.48%, rss=107768 kB (+60), threads=46 (health:122) 2018-05-08 09:28:05,441-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:28:05,469-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 09:28:05,470-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=107832 kB (+64), threads=46 (health:122) 2018-05-08 09:29:05,471-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:29:05,504-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:29:05,504-0500 DEBUG (health) [health] user=1.27%, sys=0.48%, rss=107976 kB (+144), threads=46 (health:122) 2018-05-08 09:30:05,505-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:30:05,534-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:30:05,536-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=108112 kB (+136), threads=46 (health:122) 2018-05-08 09:31:05,536-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:31:05,565-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:31:05,565-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=108724 kB (+612), threads=46 (health:122) 2018-05-08 09:32:05,566-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:32:05,595-0500 DEBUG (health) [health] Collected 122 objects (health:97) 2018-05-08 09:32:05,595-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=108840 kB (+116), threads=46 (health:122) 2018-05-08 09:33:05,596-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:33:05,624-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:33:05,625-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=108936 kB (+96), threads=46 (health:122) 2018-05-08 09:34:05,625-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:34:05,654-0500 DEBUG (health) [health] Collected 131 objects (health:97) 2018-05-08 09:34:05,655-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=109072 kB (+136), threads=46 (health:122) 2018-05-08 09:35:05,655-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:35:05,683-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:35:05,684-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=109148 kB (+76), threads=46 (health:122) 2018-05-08 09:36:05,685-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:36:05,714-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:36:05,714-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=109512 kB (+364), threads=46 (health:122) 2018-05-08 09:37:05,715-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:37:05,743-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:37:05,745-0500 DEBUG (health) [health] user=1.27%, sys=0.47%, rss=109620 kB (+108), threads=46 (health:122) 2018-05-08 09:38:05,746-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:38:05,774-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 09:38:05,775-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=109968 kB (+348), threads=46 (health:122) 2018-05-08 09:39:05,776-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:39:05,808-0500 DEBUG (health) [health] Collected 112 objects (health:97) 2018-05-08 09:39:05,808-0500 DEBUG (health) [health] user=1.28%, sys=0.52%, rss=110088 kB (+120), threads=46 (health:122) 2018-05-08 09:40:05,809-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:40:05,838-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 09:40:05,839-0500 DEBUG (health) [health] user=1.27%, sys=0.42%, rss=110156 kB (+68), threads=46 (health:122) 2018-05-08 09:41:05,839-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:41:05,868-0500 DEBUG (health) [health] Collected 76 objects (health:97) 2018-05-08 09:41:05,868-0500 DEBUG (health) [health] user=1.20%, sys=0.50%, rss=110512 kB (+356), threads=46 (health:122) 2018-05-08 09:42:05,869-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:42:05,898-0500 DEBUG (health) [health] Collected 143 objects (health:97) 2018-05-08 09:42:05,898-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=110608 kB (+96), threads=46 (health:122) 2018-05-08 09:43:05,899-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:43:05,927-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:43:05,928-0500 DEBUG (health) [health] user=1.20%, sys=0.52%, rss=110704 kB (+96), threads=46 (health:122) 2018-05-08 09:44:05,929-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:44:05,957-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:44:05,958-0500 DEBUG (health) [health] user=1.22%, sys=0.57%, rss=111044 kB (+340), threads=46 (health:122) 2018-05-08 09:45:05,958-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:45:05,987-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:45:05,988-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=111160 kB (+116), threads=46 (health:122) 2018-05-08 09:46:05,988-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:46:06,017-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:46:06,018-0500 DEBUG (health) [health] user=1.25%, sys=0.52%, rss=111256 kB (+96), threads=46 (health:122) 2018-05-08 09:47:06,019-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:47:06,047-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:47:06,048-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=111576 kB (+320), threads=46 (health:122) 2018-05-08 09:48:06,048-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:48:06,078-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:48:06,078-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=111684 kB (+108), threads=46 (health:122) 2018-05-08 09:49:06,079-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:49:06,108-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:49:06,108-0500 DEBUG (health) [health] user=1.27%, sys=0.53%, rss=111796 kB (+112), threads=46 (health:122) 2018-05-08 09:50:06,109-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:50:06,137-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 09:50:06,138-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=112136 kB (+340), threads=46 (health:122) 2018-05-08 09:51:06,139-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:51:06,167-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:51:06,168-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=112232 kB (+96), threads=46 (health:122) 2018-05-08 09:52:06,169-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:52:06,197-0500 DEBUG (health) [health] Collected 146 objects (health:97) 2018-05-08 09:52:06,198-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=112332 kB (+100), threads=46 (health:122) 2018-05-08 09:53:06,198-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:53:06,227-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:53:06,228-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=112676 kB (+344), threads=46 (health:122) 2018-05-08 09:54:06,228-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:54:06,257-0500 DEBUG (health) [health] Collected 119 objects (health:97) 2018-05-08 09:54:06,258-0500 DEBUG (health) [health] user=1.25%, sys=0.53%, rss=112780 kB (+104), threads=46 (health:122) 2018-05-08 09:55:06,259-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:55:06,288-0500 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 09:55:06,289-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=113196 kB (+416), threads=46 (health:122) 2018-05-08 09:56:06,290-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:56:06,325-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 09:56:06,326-0500 DEBUG (health) [health] user=1.30%, sys=0.48%, rss=113368 kB (+172), threads=46 (health:122) 2018-05-08 09:57:06,326-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:57:06,357-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 09:57:06,359-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=113828 kB (+460), threads=46 (health:122) 2018-05-08 09:58:06,359-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:58:06,389-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 09:58:06,389-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=114324 kB (+496), threads=46 (health:122) 2018-05-08 09:59:06,390-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:59:06,420-0500 DEBUG (health) [health] Collected 59 objects (health:97) 2018-05-08 09:59:06,421-0500 DEBUG (health) [health] user=1.25%, sys=0.55%, rss=114512 kB (+188), threads=46 (health:122) 2018-05-08 10:00:06,421-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:00:06,451-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:00:06,453-0500 DEBUG (health) [health] user=1.30%, sys=0.48%, rss=114932 kB (+420), threads=46 (health:122) 2018-05-08 10:01:06,453-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:01:06,483-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 10:01:06,484-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=115368 kB (+436), threads=46 (health:122) 2018-05-08 10:02:06,485-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:02:06,515-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:02:06,515-0500 DEBUG (health) [health] user=1.30%, sys=0.47%, rss=115572 kB (+204), threads=46 (health:122) 2018-05-08 10:03:06,516-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:03:06,546-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 10:03:06,547-0500 DEBUG (health) [health] user=1.37%, sys=0.50%, rss=116052 kB (+480), threads=46 (health:122) 2018-05-08 10:04:06,549-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:04:06,579-0500 DEBUG (health) [health] Collected 41 objects (health:97) 2018-05-08 10:04:06,579-0500 DEBUG (health) [health] user=1.27%, sys=0.53%, rss=116504 kB (+452), threads=46 (health:122) 2018-05-08 10:05:06,580-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:05:06,610-0500 DEBUG (health) [health] Collected 19 objects (health:97) 2018-05-08 10:05:06,611-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=116704 kB (+200), threads=46 (health:122) 2018-05-08 10:06:06,611-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:06:06,641-0500 DEBUG (health) [health] Collected 22 objects (health:97) 2018-05-08 10:06:06,642-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=116896 kB (+192), threads=46 (health:122) 2018-05-08 10:07:06,643-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:07:06,673-0500 DEBUG (health) [health] Collected 64 objects (health:97) 2018-05-08 10:07:06,674-0500 DEBUG (health) [health] user=1.28%, sys=0.48%, rss=117600 kB (+704), threads=46 (health:122) 2018-05-08 10:08:06,674-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:08:06,703-0500 DEBUG (health) [health] Collected 19 objects (health:97) 2018-05-08 10:08:06,704-0500 DEBUG (health) [health] user=1.27%, sys=0.47%, rss=117788 kB (+188), threads=46 (health:122) 2018-05-08 10:09:06,705-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:09:06,736-0500 DEBUG (health) [health] Collected 86 objects (health:97) 2018-05-08 10:09:06,736-0500 DEBUG (health) [health] user=1.35%, sys=0.50%, rss=118280 kB (+492), threads=46 (health:122) 2018-05-08 10:10:06,737-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:10:06,767-0500 DEBUG (health) [health] Collected 41 objects (health:97) 2018-05-08 10:10:06,769-0500 DEBUG (health) [health] user=1.30%, sys=0.50%, rss=118488 kB (+208), threads=46 (health:122) 2018-05-08 10:11:06,770-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:11:06,802-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 10:11:06,803-0500 DEBUG (health) [health] user=1.30%, sys=0.53%, rss=118932 kB (+444), threads=46 (health:122) 2018-05-08 10:12:06,804-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:12:06,833-0500 DEBUG (health) [health] Collected 22 objects (health:97) 2018-05-08 10:12:06,835-0500 DEBUG (health) [health] user=1.23%, sys=0.55%, rss=119392 kB (+460), threads=46 (health:122) 2018-05-08 10:13:06,835-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:13:06,865-0500 DEBUG (health) [health] Collected 64 objects (health:97) 2018-05-08 10:13:06,867-0500 DEBUG (health) [health] user=1.28%, sys=0.50%, rss=119828 kB (+436), threads=46 (health:122) 2018-05-08 10:14:06,868-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:14:06,898-0500 DEBUG (health) [health] Collected 46 objects (health:97) 2018-05-08 10:14:06,899-0500 DEBUG (health) [health] user=1.32%, sys=0.50%, rss=120092 kB (+264), threads=46 (health:122) 2018-05-08 10:15:06,900-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:15:06,930-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 10:15:06,932-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=120500 kB (+408), threads=46 (health:122) 2018-05-08 10:16:06,932-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:16:06,964-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 10:16:06,965-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=120692 kB (+192), threads=46 (health:122) 2018-05-08 10:17:06,966-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:17:06,996-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:17:06,997-0500 DEBUG (health) [health] user=1.30%, sys=0.45%, rss=121108 kB (+416), threads=46 (health:122) 2018-05-08 10:18:06,998-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:18:07,029-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:18:07,029-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=121568 kB (+460), threads=46 (health:122) 2018-05-08 10:19:07,030-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:19:07,061-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 10:19:07,062-0500 DEBUG (health) [health] user=1.25%, sys=0.55%, rss=121760 kB (+192), threads=46 (health:122) 2018-05-08 10:20:07,062-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:20:07,094-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 10:20:07,096-0500 DEBUG (health) [health] user=1.25%, sys=0.53%, rss=122320 kB (+560), threads=46 (health:122) 2018-05-08 10:21:07,096-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:21:07,128-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 10:21:07,129-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=122704 kB (+384), threads=46 (health:122) 2018-05-08 10:22:07,130-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:22:07,160-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 10:22:07,161-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=122852 kB (+148), threads=46 (health:122) 2018-05-08 10:23:07,162-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:23:07,193-0500 DEBUG (health) [health] Collected 21 objects (health:97) 2018-05-08 10:23:07,194-0500 DEBUG (health) [health] user=1.27%, sys=0.50%, rss=123244 kB (+392), threads=46 (health:122) 2018-05-08 10:24:07,194-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:24:07,226-0500 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 10:24:07,226-0500 DEBUG (health) [health] user=1.27%, sys=0.57%, rss=123404 kB (+160), threads=46 (health:122) 2018-05-08 10:25:07,227-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:25:07,258-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 10:25:07,259-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=123804 kB (+400), threads=46 (health:122) 2018-05-08 10:26:07,260-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:26:07,290-0500 DEBUG (health) [health] Collected 24 objects (health:97) 2018-05-08 10:26:07,293-0500 DEBUG (health) [health] user=1.28%, sys=0.48%, rss=126136 kB (+2332), threads=46 (health:122) -- Chris Adams <cma@cmadams.net>
On Tue, May 8, 2018 at 6:30 PM Chris Adams <cma@cmadams.net> wrote:
Here's the log after the first couple of hours. It is growing pretty steadliy.
2018-05-08 07:49:02,422-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:49:02,453-0500 DEBUG (health) [health] Collected 1307 objects (health:97) 2018-05-08 07:49:02,454-0500 DEBUG (health) [health] user=3.83%, sys=1.27%, rss=79344 kB (+14664), threads=46 (health:122)
We can see that we don't have an issue with noncollectable objects (otherwise you would see list of these objects on every check). Lets see if this continue to grow to unreasonable amount in the next days. Enabling the monitor this on another host which does not run the hosted engine would be useful for reference. Please open a vdsm bug for this, and add enough details on the setup so we can reproduce this in the lab. Nir
2018-05-08 07:50:02,454-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:50:02,479-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 07:50:02,480-0500 DEBUG (health) [health] user=1.18%, sys=0.47%, rss=85468 kB (+6124), threads=46 (health:122) 2018-05-08 07:51:02,481-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:51:02,507-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 07:51:02,507-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=87716 kB (+2248), threads=46 (health:122) 2018-05-08 07:52:02,508-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:52:02,534-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 07:52:02,535-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=90176 kB (+2460), threads=46 (health:122) 2018-05-08 07:53:02,536-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:53:02,561-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 07:53:02,561-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=90332 kB (+156), threads=46 (health:122) 2018-05-08 07:54:02,562-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:54:02,589-0500 DEBUG (health) [health] Collected 110 objects (health:97) 2018-05-08 07:54:02,589-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=90472 kB (+140), threads=46 (health:122) 2018-05-08 07:55:02,590-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:55:02,615-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 07:55:02,616-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=90560 kB (+88), threads=46 (health:122) 2018-05-08 07:56:02,616-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:56:02,645-0500 DEBUG (health) [health] Collected 90 objects (health:97) 2018-05-08 07:56:02,645-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=90944 kB (+384), threads=46 (health:122) 2018-05-08 07:57:02,646-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:57:02,672-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 07:57:02,673-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=91076 kB (+132), threads=46 (health:122) 2018-05-08 07:58:02,674-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:58:02,700-0500 DEBUG (health) [health] Collected 54 objects (health:97) 2018-05-08 07:58:02,703-0500 DEBUG (health) [health] user=1.22%, sys=0.42%, rss=91500 kB (+424), threads=46 (health:122) 2018-05-08 07:59:02,703-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 07:59:02,730-0500 DEBUG (health) [health] Collected 110 objects (health:97) 2018-05-08 07:59:02,730-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=91584 kB (+84), threads=46 (health:122) 2018-05-08 08:00:02,731-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:00:02,757-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:00:02,759-0500 DEBUG (health) [health] user=1.25%, sys=0.43%, rss=91696 kB (+112), threads=46 (health:122) 2018-05-08 08:01:02,760-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:01:02,789-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:01:02,790-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=91760 kB (+64), threads=46 (health:122) 2018-05-08 08:02:02,791-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:02:02,822-0500 DEBUG (health) [health] Collected 90 objects (health:97) 2018-05-08 08:02:02,823-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=92084 kB (+324), threads=46 (health:122) 2018-05-08 08:03:02,824-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:03:02,851-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 08:03:02,851-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=92308 kB (+224), threads=46 (health:122) 2018-05-08 08:04:02,853-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:04:02,891-0500 DEBUG (health) [health] Collected 99 objects (health:97) 2018-05-08 08:04:02,892-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=92652 kB (+344), threads=46 (health:122) 2018-05-08 08:05:02,893-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:05:02,921-0500 DEBUG (health) [health] Collected 51 objects (health:97) 2018-05-08 08:05:02,922-0500 DEBUG (health) [health] user=1.23%, sys=0.43%, rss=92772 kB (+120), threads=46 (health:122) 2018-05-08 08:06:02,922-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:06:02,948-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 08:06:02,949-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=92868 kB (+96), threads=46 (health:122) 2018-05-08 08:07:02,949-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:07:02,976-0500 DEBUG (health) [health] Collected 87 objects (health:97) 2018-05-08 08:07:02,979-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=93236 kB (+368), threads=46 (health:122) 2018-05-08 08:08:02,984-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:08:03,019-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 08:08:03,020-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=93436 kB (+200), threads=46 (health:122) 2018-05-08 08:09:03,021-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:09:03,048-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:09:03,048-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=93776 kB (+340), threads=46 (health:122) 2018-05-08 08:10:03,049-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:10:03,088-0500 DEBUG (health) [health] Collected 56 objects (health:97) 2018-05-08 08:10:03,088-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=93836 kB (+60), threads=46 (health:122) 2018-05-08 08:11:03,089-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:11:03,115-0500 DEBUG (health) [health] Collected 54 objects (health:97) 2018-05-08 08:11:03,116-0500 DEBUG (health) [health] user=1.15%, sys=0.48%, rss=93936 kB (+100), threads=46 (health:122) 2018-05-08 08:12:03,117-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:12:03,142-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:12:03,143-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=94052 kB (+116), threads=46 (health:122) 2018-05-08 08:13:03,143-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:13:03,169-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 08:13:03,169-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=94200 kB (+148), threads=46 (health:122) 2018-05-08 08:14:03,170-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:14:03,199-0500 DEBUG (health) [health] Collected 131 objects (health:97) 2018-05-08 08:14:03,200-0500 DEBUG (health) [health] user=1.27%, sys=0.53%, rss=94576 kB (+376), threads=46 (health:122) 2018-05-08 08:15:03,200-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:15:03,228-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:15:03,232-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=94660 kB (+84), threads=46 (health:122) 2018-05-08 08:16:03,232-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:16:03,258-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 08:16:03,261-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=95012 kB (+352), threads=46 (health:122) 2018-05-08 08:17:03,262-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:17:03,293-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 08:17:03,294-0500 DEBUG (health) [health] user=1.18%, sys=0.50%, rss=95076 kB (+64), threads=46 (health:122) 2018-05-08 08:18:03,295-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:18:03,320-0500 DEBUG (health) [health] Collected 78 objects (health:97) 2018-05-08 08:18:03,323-0500 DEBUG (health) [health] user=1.20%, sys=0.45%, rss=95168 kB (+92), threads=46 (health:122) 2018-05-08 08:19:03,324-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:19:03,350-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 08:19:03,350-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=95248 kB (+80), threads=46 (health:122) 2018-05-08 08:20:03,351-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:20:03,376-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 08:20:03,377-0500 DEBUG (health) [health] user=1.28%, sys=0.47%, rss=95272 kB (+24), threads=46 (health:122) 2018-05-08 08:21:03,377-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:21:03,420-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 08:21:03,420-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=95664 kB (+392), threads=46 (health:122) 2018-05-08 08:22:03,421-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:22:03,447-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:22:03,449-0500 DEBUG (health) [health] user=1.27%, sys=0.52%, rss=95800 kB (+136), threads=46 (health:122) 2018-05-08 08:23:03,449-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:23:03,477-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 08:23:03,477-0500 DEBUG (health) [health] user=1.28%, sys=0.52%, rss=96216 kB (+416), threads=46 (health:122) 2018-05-08 08:24:03,478-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:24:03,505-0500 DEBUG (health) [health] Collected 87 objects (health:97) 2018-05-08 08:24:03,505-0500 DEBUG (health) [health] user=1.35%, sys=0.52%, rss=96748 kB (+532), threads=46 (health:122) 2018-05-08 08:25:03,506-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:25:03,533-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:25:03,534-0500 DEBUG (health) [health] user=1.28%, sys=0.47%, rss=96884 kB (+136), threads=46 (health:122) 2018-05-08 08:26:03,534-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:26:03,560-0500 DEBUG (health) [health] Collected 63 objects (health:97) 2018-05-08 08:26:03,561-0500 DEBUG (health) [health] user=1.30%, sys=0.43%, rss=97640 kB (+756), threads=46 (health:122) 2018-05-08 08:27:03,562-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:27:03,591-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 08:27:03,593-0500 DEBUG (health) [health] user=1.25%, sys=0.52%, rss=97792 kB (+152), threads=46 (health:122) 2018-05-08 08:28:03,593-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:28:03,619-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 08:28:03,621-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=97904 kB (+112), threads=46 (health:122) 2018-05-08 08:29:03,621-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:29:03,648-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:29:03,648-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=97976 kB (+72), threads=46 (health:122) 2018-05-08 08:30:03,649-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:30:03,677-0500 DEBUG (health) [health] Collected 44 objects (health:97) 2018-05-08 08:30:03,677-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=98016 kB (+40), threads=46 (health:122) 2018-05-08 08:31:03,678-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:31:03,706-0500 DEBUG (health) [health] Collected 149 objects (health:97) 2018-05-08 08:31:03,706-0500 DEBUG (health) [health] user=1.23%, sys=0.42%, rss=98372 kB (+356), threads=46 (health:122) 2018-05-08 08:32:03,707-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:32:03,733-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:32:03,734-0500 DEBUG (health) [health] user=1.23%, sys=0.43%, rss=98456 kB (+84), threads=46 (health:122) 2018-05-08 08:33:03,734-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:33:03,760-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:33:03,762-0500 DEBUG (health) [health] user=1.20%, sys=0.45%, rss=98516 kB (+60), threads=46 (health:122) 2018-05-08 08:34:03,762-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:34:03,789-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 08:34:03,789-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=98584 kB (+68), threads=46 (health:122) 2018-05-08 08:35:03,790-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:35:03,816-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:35:03,818-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=98632 kB (+48), threads=46 (health:122) 2018-05-08 08:36:03,818-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:36:03,851-0500 DEBUG (health) [health] Collected 66 objects (health:97) 2018-05-08 08:36:03,852-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=98944 kB (+312), threads=46 (health:122) 2018-05-08 08:37:03,854-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:37:03,881-0500 DEBUG (health) [health] Collected 188 objects (health:97) 2018-05-08 08:37:03,882-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=98992 kB (+48), threads=46 (health:122) 2018-05-08 08:38:03,882-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:38:03,911-0500 DEBUG (health) [health] Collected 167 objects (health:97) 2018-05-08 08:38:03,911-0500 DEBUG (health) [health] user=1.20%, sys=0.47%, rss=99076 kB (+84), threads=46 (health:122) 2018-05-08 08:39:03,912-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:39:03,939-0500 DEBUG (health) [health] Collected 111 objects (health:97) 2018-05-08 08:39:03,939-0500 DEBUG (health) [health] user=1.20%, sys=0.52%, rss=99408 kB (+332), threads=46 (health:122) 2018-05-08 08:40:03,940-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:40:03,967-0500 DEBUG (health) [health] Collected 107 objects (health:97) 2018-05-08 08:40:03,968-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=99492 kB (+84), threads=46 (health:122) 2018-05-08 08:41:03,969-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:41:03,997-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 08:41:03,997-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=99596 kB (+104), threads=46 (health:122) 2018-05-08 08:42:03,998-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:42:04,026-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 08:42:04,027-0500 DEBUG (health) [health] user=1.23%, sys=0.43%, rss=99684 kB (+88), threads=46 (health:122) 2018-05-08 08:43:04,027-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:43:04,055-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 08:43:04,056-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=100052 kB (+368), threads=46 (health:122) 2018-05-08 08:44:04,057-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:44:04,100-0500 DEBUG (health) [health] Collected 83 objects (health:97) 2018-05-08 08:44:04,101-0500 DEBUG (health) [health] user=1.28%, sys=0.45%, rss=100168 kB (+116), threads=46 (health:122) 2018-05-08 08:45:04,101-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:45:04,129-0500 DEBUG (health) [health] Collected 19 objects (health:97) 2018-05-08 08:45:04,129-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=100244 kB (+76), threads=46 (health:122) 2018-05-08 08:46:04,130-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:46:04,157-0500 DEBUG (health) [health] Collected 104 objects (health:97) 2018-05-08 08:46:04,157-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=100340 kB (+96), threads=46 (health:122) 2018-05-08 08:47:04,158-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:47:04,186-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 08:47:04,186-0500 DEBUG (health) [health] user=1.27%, sys=0.52%, rss=100672 kB (+332), threads=46 (health:122) 2018-05-08 08:48:04,187-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:48:04,216-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:48:04,216-0500 DEBUG (health) [health] user=1.18%, sys=0.47%, rss=100740 kB (+68), threads=46 (health:122) 2018-05-08 08:49:04,217-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:49:04,246-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 08:49:04,246-0500 DEBUG (health) [health] user=1.32%, sys=0.50%, rss=100828 kB (+88), threads=46 (health:122) 2018-05-08 08:50:04,247-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:50:04,275-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:50:04,275-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=101196 kB (+368), threads=46 (health:122) 2018-05-08 08:51:04,276-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:51:04,305-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 08:51:04,322-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=101284 kB (+88), threads=46 (health:122) 2018-05-08 08:52:04,323-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:52:04,353-0500 DEBUG (health) [health] Collected 122 objects (health:97) 2018-05-08 08:52:04,353-0500 DEBUG (health) [health] user=1.22%, sys=0.45%, rss=101632 kB (+348), threads=46 (health:122) 2018-05-08 08:53:04,354-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:53:04,387-0500 DEBUG (health) [health] Collected 93 objects (health:97) 2018-05-08 08:53:04,387-0500 DEBUG (health) [health] user=1.30%, sys=0.48%, rss=101724 kB (+92), threads=46 (health:122) 2018-05-08 08:54:04,388-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:54:04,416-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 08:54:04,416-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=101800 kB (+76), threads=46 (health:122) 2018-05-08 08:55:04,421-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:55:04,449-0500 DEBUG (health) [health] Collected 24 objects (health:97) 2018-05-08 08:55:04,449-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=101848 kB (+48), threads=46 (health:122) 2018-05-08 08:56:04,450-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:56:04,479-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:56:04,480-0500 DEBUG (health) [health] user=1.22%, sys=0.47%, rss=102192 kB (+344), threads=46 (health:122) 2018-05-08 08:57:04,481-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:57:04,509-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 08:57:04,510-0500 DEBUG (health) [health] user=1.27%, sys=0.45%, rss=102256 kB (+64), threads=46 (health:122) 2018-05-08 08:58:04,510-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:58:04,555-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 08:58:04,556-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=102380 kB (+124), threads=46 (health:122) 2018-05-08 08:59:04,557-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 08:59:04,585-0500 DEBUG (health) [health] Collected 115 objects (health:97) 2018-05-08 08:59:04,586-0500 DEBUG (health) [health] user=1.30%, sys=0.53%, rss=102796 kB (+416), threads=46 (health:122) 2018-05-08 09:00:04,586-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:00:04,616-0500 DEBUG (health) [health] Collected 108 objects (health:97) 2018-05-08 09:00:04,617-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=102944 kB (+148), threads=46 (health:122) 2018-05-08 09:01:04,617-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:01:04,651-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 09:01:04,652-0500 DEBUG (health) [health] user=1.27%, sys=0.43%, rss=103048 kB (+104), threads=46 (health:122) 2018-05-08 09:02:04,653-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:02:04,682-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 09:02:04,683-0500 DEBUG (health) [health] user=1.28%, sys=0.42%, rss=103388 kB (+340), threads=46 (health:122) 2018-05-08 09:03:04,684-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:03:04,713-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 09:03:04,714-0500 DEBUG (health) [health] user=1.25%, sys=0.43%, rss=103488 kB (+100), threads=46 (health:122) 2018-05-08 09:04:04,715-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:04:04,743-0500 DEBUG (health) [health] Collected 64 objects (health:97) 2018-05-08 09:04:04,744-0500 DEBUG (health) [health] user=1.27%, sys=0.50%, rss=103592 kB (+104), threads=46 (health:122) 2018-05-08 09:05:04,744-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:05:04,775-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 09:05:04,776-0500 DEBUG (health) [health] user=1.22%, sys=0.45%, rss=103912 kB (+320), threads=46 (health:122) 2018-05-08 09:06:04,776-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:06:04,804-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 09:06:04,804-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=104016 kB (+104), threads=46 (health:122) 2018-05-08 09:07:04,805-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:07:04,832-0500 DEBUG (health) [health] Collected 104 objects (health:97) 2018-05-08 09:07:04,833-0500 DEBUG (health) [health] user=1.27%, sys=0.45%, rss=104104 kB (+88), threads=46 (health:122) 2018-05-08 09:08:04,834-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:08:04,861-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:08:04,862-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=104200 kB (+96), threads=46 (health:122) 2018-05-08 09:09:04,863-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:09:04,890-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:09:04,891-0500 DEBUG (health) [health] user=1.28%, sys=0.53%, rss=104584 kB (+384), threads=46 (health:122) 2018-05-08 09:10:04,891-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:10:04,920-0500 DEBUG (health) [health] Collected 111 objects (health:97) 2018-05-08 09:10:04,921-0500 DEBUG (health) [health] user=1.27%, sys=0.50%, rss=104692 kB (+108), threads=46 (health:122) 2018-05-08 09:11:04,922-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:11:04,950-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 09:11:04,951-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=104744 kB (+52), threads=46 (health:122) 2018-05-08 09:12:04,951-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:12:04,980-0500 DEBUG (health) [health] Collected 52 objects (health:97) 2018-05-08 09:12:04,997-0500 DEBUG (health) [health] user=1.20%, sys=0.52%, rss=105092 kB (+348), threads=46 (health:122) 2018-05-08 09:13:04,998-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:13:05,026-0500 DEBUG (health) [health] Collected 67 objects (health:97) 2018-05-08 09:13:05,027-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=105180 kB (+88), threads=46 (health:122) 2018-05-08 09:14:05,027-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:14:05,055-0500 DEBUG (health) [health] Collected 131 objects (health:97) 2018-05-08 09:14:05,056-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=105536 kB (+356), threads=46 (health:122) 2018-05-08 09:15:05,056-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:15:05,085-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:15:05,086-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=105636 kB (+100), threads=46 (health:122) 2018-05-08 09:16:05,087-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:16:05,115-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 09:16:05,116-0500 DEBUG (health) [health] user=1.20%, sys=0.47%, rss=105724 kB (+88), threads=46 (health:122) 2018-05-08 09:17:05,116-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:17:05,144-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:17:05,146-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=106036 kB (+312), threads=46 (health:122) 2018-05-08 09:18:05,146-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:18:05,174-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:18:05,175-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=106104 kB (+68), threads=46 (health:122) 2018-05-08 09:19:05,175-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:19:05,204-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:19:05,204-0500 DEBUG (health) [health] user=1.20%, sys=0.53%, rss=106228 kB (+124), threads=46 (health:122) 2018-05-08 09:20:05,205-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:20:05,233-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 09:20:05,234-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=106292 kB (+64), threads=46 (health:122) 2018-05-08 09:21:05,235-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:21:05,263-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 09:21:05,263-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=106676 kB (+384), threads=46 (health:122) 2018-05-08 09:22:05,264-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:22:05,292-0500 DEBUG (health) [health] Collected 143 objects (health:97) 2018-05-08 09:22:05,293-0500 DEBUG (health) [health] user=1.27%, sys=0.45%, rss=107008 kB (+332), threads=46 (health:122) 2018-05-08 09:23:05,293-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:23:05,322-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 09:23:05,324-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=107072 kB (+64), threads=46 (health:122) 2018-05-08 09:24:05,324-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:24:05,353-0500 DEBUG (health) [health] Collected 127 objects (health:97) 2018-05-08 09:24:05,353-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=107192 kB (+120), threads=46 (health:122) 2018-05-08 09:25:05,354-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:25:05,381-0500 DEBUG (health) [health] Collected 88 objects (health:97) 2018-05-08 09:25:05,383-0500 DEBUG (health) [health] user=1.18%, sys=0.53%, rss=107356 kB (+164), threads=46 (health:122) 2018-05-08 09:26:05,383-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:26:05,411-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 09:26:05,412-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=107708 kB (+352), threads=46 (health:122) 2018-05-08 09:27:05,412-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:27:05,440-0500 DEBUG (health) [health] Collected 125 objects (health:97) 2018-05-08 09:27:05,441-0500 DEBUG (health) [health] user=1.27%, sys=0.48%, rss=107768 kB (+60), threads=46 (health:122) 2018-05-08 09:28:05,441-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:28:05,469-0500 DEBUG (health) [health] Collected 85 objects (health:97) 2018-05-08 09:28:05,470-0500 DEBUG (health) [health] user=1.25%, sys=0.45%, rss=107832 kB (+64), threads=46 (health:122) 2018-05-08 09:29:05,471-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:29:05,504-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:29:05,504-0500 DEBUG (health) [health] user=1.27%, sys=0.48%, rss=107976 kB (+144), threads=46 (health:122) 2018-05-08 09:30:05,505-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:30:05,534-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:30:05,536-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=108112 kB (+136), threads=46 (health:122) 2018-05-08 09:31:05,536-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:31:05,565-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:31:05,565-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=108724 kB (+612), threads=46 (health:122) 2018-05-08 09:32:05,566-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:32:05,595-0500 DEBUG (health) [health] Collected 122 objects (health:97) 2018-05-08 09:32:05,595-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=108840 kB (+116), threads=46 (health:122) 2018-05-08 09:33:05,596-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:33:05,624-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:33:05,625-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=108936 kB (+96), threads=46 (health:122) 2018-05-08 09:34:05,625-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:34:05,654-0500 DEBUG (health) [health] Collected 131 objects (health:97) 2018-05-08 09:34:05,655-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=109072 kB (+136), threads=46 (health:122) 2018-05-08 09:35:05,655-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:35:05,683-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:35:05,684-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=109148 kB (+76), threads=46 (health:122) 2018-05-08 09:36:05,685-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:36:05,714-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:36:05,714-0500 DEBUG (health) [health] user=1.23%, sys=0.48%, rss=109512 kB (+364), threads=46 (health:122) 2018-05-08 09:37:05,715-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:37:05,743-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:37:05,745-0500 DEBUG (health) [health] user=1.27%, sys=0.47%, rss=109620 kB (+108), threads=46 (health:122) 2018-05-08 09:38:05,746-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:38:05,774-0500 DEBUG (health) [health] Collected 106 objects (health:97) 2018-05-08 09:38:05,775-0500 DEBUG (health) [health] user=1.22%, sys=0.48%, rss=109968 kB (+348), threads=46 (health:122) 2018-05-08 09:39:05,776-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:39:05,808-0500 DEBUG (health) [health] Collected 112 objects (health:97) 2018-05-08 09:39:05,808-0500 DEBUG (health) [health] user=1.28%, sys=0.52%, rss=110088 kB (+120), threads=46 (health:122) 2018-05-08 09:40:05,809-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:40:05,838-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 09:40:05,839-0500 DEBUG (health) [health] user=1.27%, sys=0.42%, rss=110156 kB (+68), threads=46 (health:122) 2018-05-08 09:41:05,839-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:41:05,868-0500 DEBUG (health) [health] Collected 76 objects (health:97) 2018-05-08 09:41:05,868-0500 DEBUG (health) [health] user=1.20%, sys=0.50%, rss=110512 kB (+356), threads=46 (health:122) 2018-05-08 09:42:05,869-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:42:05,898-0500 DEBUG (health) [health] Collected 143 objects (health:97) 2018-05-08 09:42:05,898-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=110608 kB (+96), threads=46 (health:122) 2018-05-08 09:43:05,899-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:43:05,927-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:43:05,928-0500 DEBUG (health) [health] user=1.20%, sys=0.52%, rss=110704 kB (+96), threads=46 (health:122) 2018-05-08 09:44:05,929-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:44:05,957-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:44:05,958-0500 DEBUG (health) [health] user=1.22%, sys=0.57%, rss=111044 kB (+340), threads=46 (health:122) 2018-05-08 09:45:05,958-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:45:05,987-0500 DEBUG (health) [health] Collected 91 objects (health:97) 2018-05-08 09:45:05,988-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=111160 kB (+116), threads=46 (health:122) 2018-05-08 09:46:05,988-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:46:06,017-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:46:06,018-0500 DEBUG (health) [health] user=1.25%, sys=0.52%, rss=111256 kB (+96), threads=46 (health:122) 2018-05-08 09:47:06,019-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:47:06,047-0500 DEBUG (health) [health] Collected 128 objects (health:97) 2018-05-08 09:47:06,048-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=111576 kB (+320), threads=46 (health:122) 2018-05-08 09:48:06,048-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:48:06,078-0500 DEBUG (health) [health] Collected 109 objects (health:97) 2018-05-08 09:48:06,078-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=111684 kB (+108), threads=46 (health:122) 2018-05-08 09:49:06,079-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:49:06,108-0500 DEBUG (health) [health] Collected 113 objects (health:97) 2018-05-08 09:49:06,108-0500 DEBUG (health) [health] user=1.27%, sys=0.53%, rss=111796 kB (+112), threads=46 (health:122) 2018-05-08 09:50:06,109-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:50:06,137-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 09:50:06,138-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=112136 kB (+340), threads=46 (health:122) 2018-05-08 09:51:06,139-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:51:06,167-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:51:06,168-0500 DEBUG (health) [health] user=1.22%, sys=0.50%, rss=112232 kB (+96), threads=46 (health:122) 2018-05-08 09:52:06,169-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:52:06,197-0500 DEBUG (health) [health] Collected 146 objects (health:97) 2018-05-08 09:52:06,198-0500 DEBUG (health) [health] user=1.23%, sys=0.47%, rss=112332 kB (+100), threads=46 (health:122) 2018-05-08 09:53:06,198-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:53:06,227-0500 DEBUG (health) [health] Collected 79 objects (health:97) 2018-05-08 09:53:06,228-0500 DEBUG (health) [health] user=1.20%, sys=0.48%, rss=112676 kB (+344), threads=46 (health:122) 2018-05-08 09:54:06,228-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:54:06,257-0500 DEBUG (health) [health] Collected 119 objects (health:97) 2018-05-08 09:54:06,258-0500 DEBUG (health) [health] user=1.25%, sys=0.53%, rss=112780 kB (+104), threads=46 (health:122) 2018-05-08 09:55:06,259-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:55:06,288-0500 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 09:55:06,289-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=113196 kB (+416), threads=46 (health:122) 2018-05-08 09:56:06,290-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:56:06,325-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 09:56:06,326-0500 DEBUG (health) [health] user=1.30%, sys=0.48%, rss=113368 kB (+172), threads=46 (health:122) 2018-05-08 09:57:06,326-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:57:06,357-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 09:57:06,359-0500 DEBUG (health) [health] user=1.25%, sys=0.47%, rss=113828 kB (+460), threads=46 (health:122) 2018-05-08 09:58:06,359-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:58:06,389-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 09:58:06,389-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=114324 kB (+496), threads=46 (health:122) 2018-05-08 09:59:06,390-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 09:59:06,420-0500 DEBUG (health) [health] Collected 59 objects (health:97) 2018-05-08 09:59:06,421-0500 DEBUG (health) [health] user=1.25%, sys=0.55%, rss=114512 kB (+188), threads=46 (health:122) 2018-05-08 10:00:06,421-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:00:06,451-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:00:06,453-0500 DEBUG (health) [health] user=1.30%, sys=0.48%, rss=114932 kB (+420), threads=46 (health:122) 2018-05-08 10:01:06,453-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:01:06,483-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 10:01:06,484-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=115368 kB (+436), threads=46 (health:122) 2018-05-08 10:02:06,485-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:02:06,515-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:02:06,515-0500 DEBUG (health) [health] user=1.30%, sys=0.47%, rss=115572 kB (+204), threads=46 (health:122) 2018-05-08 10:03:06,516-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:03:06,546-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 10:03:06,547-0500 DEBUG (health) [health] user=1.37%, sys=0.50%, rss=116052 kB (+480), threads=46 (health:122) 2018-05-08 10:04:06,549-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:04:06,579-0500 DEBUG (health) [health] Collected 41 objects (health:97) 2018-05-08 10:04:06,579-0500 DEBUG (health) [health] user=1.27%, sys=0.53%, rss=116504 kB (+452), threads=46 (health:122) 2018-05-08 10:05:06,580-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:05:06,610-0500 DEBUG (health) [health] Collected 19 objects (health:97) 2018-05-08 10:05:06,611-0500 DEBUG (health) [health] user=1.25%, sys=0.48%, rss=116704 kB (+200), threads=46 (health:122) 2018-05-08 10:06:06,611-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:06:06,641-0500 DEBUG (health) [health] Collected 22 objects (health:97) 2018-05-08 10:06:06,642-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=116896 kB (+192), threads=46 (health:122) 2018-05-08 10:07:06,643-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:07:06,673-0500 DEBUG (health) [health] Collected 64 objects (health:97) 2018-05-08 10:07:06,674-0500 DEBUG (health) [health] user=1.28%, sys=0.48%, rss=117600 kB (+704), threads=46 (health:122) 2018-05-08 10:08:06,674-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:08:06,703-0500 DEBUG (health) [health] Collected 19 objects (health:97) 2018-05-08 10:08:06,704-0500 DEBUG (health) [health] user=1.27%, sys=0.47%, rss=117788 kB (+188), threads=46 (health:122) 2018-05-08 10:09:06,705-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:09:06,736-0500 DEBUG (health) [health] Collected 86 objects (health:97) 2018-05-08 10:09:06,736-0500 DEBUG (health) [health] user=1.35%, sys=0.50%, rss=118280 kB (+492), threads=46 (health:122) 2018-05-08 10:10:06,737-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:10:06,767-0500 DEBUG (health) [health] Collected 41 objects (health:97) 2018-05-08 10:10:06,769-0500 DEBUG (health) [health] user=1.30%, sys=0.50%, rss=118488 kB (+208), threads=46 (health:122) 2018-05-08 10:11:06,770-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:11:06,802-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 10:11:06,803-0500 DEBUG (health) [health] user=1.30%, sys=0.53%, rss=118932 kB (+444), threads=46 (health:122) 2018-05-08 10:12:06,804-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:12:06,833-0500 DEBUG (health) [health] Collected 22 objects (health:97) 2018-05-08 10:12:06,835-0500 DEBUG (health) [health] user=1.23%, sys=0.55%, rss=119392 kB (+460), threads=46 (health:122) 2018-05-08 10:13:06,835-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:13:06,865-0500 DEBUG (health) [health] Collected 64 objects (health:97) 2018-05-08 10:13:06,867-0500 DEBUG (health) [health] user=1.28%, sys=0.50%, rss=119828 kB (+436), threads=46 (health:122) 2018-05-08 10:14:06,868-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:14:06,898-0500 DEBUG (health) [health] Collected 46 objects (health:97) 2018-05-08 10:14:06,899-0500 DEBUG (health) [health] user=1.32%, sys=0.50%, rss=120092 kB (+264), threads=46 (health:122) 2018-05-08 10:15:06,900-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:15:06,930-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 10:15:06,932-0500 DEBUG (health) [health] user=1.25%, sys=0.50%, rss=120500 kB (+408), threads=46 (health:122) 2018-05-08 10:16:06,932-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:16:06,964-0500 DEBUG (health) [health] Collected 40 objects (health:97) 2018-05-08 10:16:06,965-0500 DEBUG (health) [health] user=1.23%, sys=0.52%, rss=120692 kB (+192), threads=46 (health:122) 2018-05-08 10:17:06,966-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:17:06,996-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:17:06,997-0500 DEBUG (health) [health] user=1.30%, sys=0.45%, rss=121108 kB (+416), threads=46 (health:122) 2018-05-08 10:18:06,998-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:18:07,029-0500 DEBUG (health) [health] Collected 37 objects (health:97) 2018-05-08 10:18:07,029-0500 DEBUG (health) [health] user=1.22%, sys=0.53%, rss=121568 kB (+460), threads=46 (health:122) 2018-05-08 10:19:07,030-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:19:07,061-0500 DEBUG (health) [health] Collected 61 objects (health:97) 2018-05-08 10:19:07,062-0500 DEBUG (health) [health] user=1.25%, sys=0.55%, rss=121760 kB (+192), threads=46 (health:122) 2018-05-08 10:20:07,062-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:20:07,094-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 10:20:07,096-0500 DEBUG (health) [health] user=1.25%, sys=0.53%, rss=122320 kB (+560), threads=46 (health:122) 2018-05-08 10:21:07,096-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:21:07,128-0500 DEBUG (health) [health] Collected 39 objects (health:97) 2018-05-08 10:21:07,129-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=122704 kB (+384), threads=46 (health:122) 2018-05-08 10:22:07,130-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:22:07,160-0500 DEBUG (health) [health] Collected 97 objects (health:97) 2018-05-08 10:22:07,161-0500 DEBUG (health) [health] user=1.22%, sys=0.52%, rss=122852 kB (+148), threads=46 (health:122) 2018-05-08 10:23:07,162-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:23:07,193-0500 DEBUG (health) [health] Collected 21 objects (health:97) 2018-05-08 10:23:07,194-0500 DEBUG (health) [health] user=1.27%, sys=0.50%, rss=123244 kB (+392), threads=46 (health:122) 2018-05-08 10:24:07,194-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:24:07,226-0500 DEBUG (health) [health] Collected 43 objects (health:97) 2018-05-08 10:24:07,226-0500 DEBUG (health) [health] user=1.27%, sys=0.57%, rss=123404 kB (+160), threads=46 (health:122) 2018-05-08 10:25:07,227-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:25:07,258-0500 DEBUG (health) [health] Collected 42 objects (health:97) 2018-05-08 10:25:07,259-0500 DEBUG (health) [health] user=1.23%, sys=0.50%, rss=123804 kB (+400), threads=46 (health:122) 2018-05-08 10:26:07,260-0500 DEBUG (health) [health] Checking health (health:90) 2018-05-08 10:26:07,290-0500 DEBUG (health) [health] Collected 24 objects (health:97) 2018-05-08 10:26:07,293-0500 DEBUG (health) [health] user=1.28%, sys=0.48%, rss=126136 kB (+2332), threads=46 (health:122)
-- Chris Adams <cma@cmadams.net>
Once upon a time, Nir Soffer <nsoffer@redhat.com> said:
Lets see if this continue to grow to unreasonable amount in the next days.
Enabling the monitor this on another host which does not run the hosted engine would be useful for reference.
Okay, I'll go enable it on the other node.
Please open a vdsm bug for this, and add enough details on the setup so we can reproduce this in the lab.
I did that a couple of weeks ago: https://bugzilla.redhat.com/show_bug.cgi?id=1570840 Please let me know what additional details could be useful - I'm guessing that this is related to something I did (since others aren't seeing it), but I'm afraid that I don't know what I did that might have been special. -- Chris Adams <cma@cmadams.net>
On Tue, May 8, 2018 at 7:10 PM Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Nir Soffer <nsoffer@redhat.com> said:
Lets see if this continue to grow to unreasonable amount in the next days.
Enabling the monitor this on another host which does not run the hosted engine would be useful for reference.
Okay, I'll go enable it on the other node.
Please open a vdsm bug for this, and add enough details on the setup so we can reproduce this in the lab.
I did that a couple of weeks ago:
Good
Please let me know what additional details could be useful - I'm guessing that this is related to something I did (since others aren't seeing it), but I'm afraid that I don't know what I did that might have been special.
Can describe your setup, so we can create a similar setup in the lab? I guess that attaching engine db dump will make this easy. Hosted engine answer file can also help. Nir
participants (3)
-
Chris Adams -
Michal Skrivanek -
Nir Soffer